borg Documentation

lsblk --fs -o +PTUUID /dev/sdz

ACTION=="add", SUBSYSTEM=="block", ENV{ID_PART_TABLE_UUID}=="", TAG+="systemd", ENV{SYSTEMD_WANTS}+="automatic-backup.service"

[Service]
Type=oneshot
ExecStart=/etc/backups/run.sh

#!/bin/bash -ue

# The udev rule is not terribly accurate and may trigger our service before
# the kernel has finished probing partitions. Sleep for a bit to ensure
# the kernel is done.
#
# This can be avoided by using a more precise udev rule, e.g. matching
# a specific hardware path and partition.
sleep 5

#
# Script configuration
#

# The backup partition is mounted there
MOUNTPOINT=/mnt/backup

# This is the location of the Borg repository
TARGET=$MOUNTPOINT/borg-backups/backup.borg

# Archive name schema
DATE=$(date --iso-8601)-$(hostname)

# This is the file that will later contain UUIDs of registered backup drives
DISKS=/etc/backups/backup.disks

# Find whether the connected block device is a backup drive
for uuid in $(lsblk --noheadings --list --output uuid)
do
        if grep --quiet --fixed-strings $uuid $DISKS; then
                break
        fi
        uuid=
done

if [ ! $uuid ]; then
        echo "No backup disk found, exiting"
        exit 0
fi

echo "Disk $uuid is a backup disk"
partition_path=/dev/disk/by-uuid/$uuid
# Mount filesystem if not already done. This assumes that if something is already
# mounted at $MOUNTPOINT, it is the backup drive. It will not find the drive if
# it was mounted somewhere else.
findmnt $MOUNTPOINT >/dev/null || mount $partition_path $MOUNTPOINT
drive=$(lsblk --inverse --noheadings --list --paths --output name $partition_path | head --lines 1)
echo "Drive path: $drive"

#
# Create backups
#

# Options for borg create
BORG_OPTS="--stats --one-file-system --compression lz4"

# Set BORG_PASSPHRASE or BORG_PASSCOMMAND somewhere around here, using export,
# if encryption is used.

# Because no one can answer these questions non-interactively, it is better to
# fail quickly instead of hanging.
export BORG_RELOCATED_REPO_ACCESS_IS_OK=no
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=no

# Log Borg version
borg --version

echo "Starting backup for $DATE"

# This is just an example, change it however you see fit
borg create $BORG_OPTS \
  --exclude root/.cache \
  --exclude var/lib/docker/devicemapper \
  $TARGET::$DATE-$$-system \
  / /boot

# /home is often a separate partition/filesystem.
# Even if it is not (add --exclude /home above), it probably makes sense
# to have /home in a separate archive.
borg create $BORG_OPTS \
  --exclude 'sh:home/*/.cache' \
  $TARGET::$DATE-$$-home \
  /home/

echo "Completed backup for $DATE"

# Just to be completely paranoid
sync

if [ -f /etc/backups/autoeject ]; then
        umount $MOUNTPOINT
        hdparm -Y $drive
fi

if [ -f /etc/backups/backup-suspend ]; then
        systemctl suspend
fi

ln -s /etc/backups/80-backup.rules /etc/udev/rules.d/80-backup.rules
ln -s /etc/backups/automatic-backup.service /etc/systemd/system/automatic-backup.service
systemctl daemon-reload
udevadm control --reload

lsblk -o+uuid,label

borg init --encryption ... /mnt/backup/borg-backups/backup.borg

systemctl start --no-block automatic-backup

journalctl -fu automatic-backup [-n number-of-lines]

Remote: borg.remote.PathNotAllowed: /home/backup/repos/web01.srv.local/pictures
  ~~~ SNIP ~~~
  Repository path not allowed

Ansible
-------

Ansible takes care of all the system-specific commands to add the user, create the
folder, install and configure software.

::

  - hosts: backup01.srv.local
    vars:
      user: backup
      group: backup
      home: /home/backup
      pool: "{{ home }}/repos"
      auth_users:
        - host: johndoe.clnt.local
          key: "{{ lookup('file', '/path/to/keys/johndoe.clnt.local.pub') }}"
        - host: web01.clnt.local
          key: "{{ lookup('file', '/path/to/keys/web01.clnt.local.pub') }}"
        - host: app01.clnt.local
          key: "{{ lookup('file', '/path/to/keys/app01.clnt.local.pub') }}"
    tasks:
    - package: name=borg state=present
    - group: name="{{ group }}" state=present
    - user: name="{{ user }}" shell=/bin/bash home="{{ home }}" createhome=yes group="{{ group }}" groups= state=present
    - file: path="{{ home }}" owner="{{ user }}" group="{{ group }}" mode=0700 state=directory
    - file: path="{{ home }}/.ssh" owner="{{ user }}" group="{{ group }}" mode=0700 state=directory
    - file: path="{{ pool }}" owner="{{ user }}" group="{{ group }}" mode=0700 state=directory
    - authorized_key: user="{{ user }}"
                      key="{{ item.key }}"
                      key_options='command="cd {{ pool }}/{{ item.host }};borg serve --restrict-to-path {{ pool }}/{{ item.host }}",restrict'
      with_items: "{{ auth_users }}"
    - file: path="{{ home }}/.ssh/authorized_keys" owner="{{ user }}" group="{{ group }}" mode=0600 state=file
    - file: path="{{ pool }}/{{ item.host }}" owner="{{ user }}" group="{{ group }}" mode=0700 state=directory
      with_items: "{{ auth_users }}"

Salt
----

This is a configuration similar to the one above, configured to be deployed with
Salt running on a Debian system.

::

  Install borg backup from pip:
    pkg.installed:
      - pkgs:
        - python3
        - python3-dev
        - python3-pip
        - python-virtualenv
        - libssl-dev
        - openssl
        - libacl1-dev
        - libacl1
        - build-essential
        - libfuse-dev
        - fuse
        - pkg-config
    pip.installed:
      - pkgs: ["borgbackup"]
      - bin_env: /usr/bin/pip3

  Setup backup user:
    user.present:
      - name: backup
      - fullname: Backup User
      - home: /home/backup
      - shell: /bin/bash
  # CAUTION!
  # If you change the ssh command= option below, it won't necessarily get pushed to the backup
  # server correctly unless you delete the ~/.ssh/authorized_keys file and re-create it!
  {% for host in backupclients %}
  Give backup access to {{host}}:
    ssh_auth.present:
      - user: backup
      - source: salt://conf/ssh-pubkeys/{{host}}-backup.id_ecdsa.pub
      - options:
        - command="cd /home/backup/repos/{{host}}; borg serve --restrict-to-path /home/backup/repos/{{host}}"
        - restrict
  {% endfor %}


Enhancements
------------

As this section only describes a simple and effective setup, it could be further
enhanced when supporting (a limited set) of client supplied commands. A wrapper
for starting `borg serve` could be written. Or borg itself could be enhanced to
autodetect it runs under SSH by checking the `SSH_ORIGINAL_COMMAND` environment
variable. This is left open for future improvements.

When extending ssh autodetection in borg no external wrapper script is necessary
and no other interpreter or application has to be deployed.

See also
--------

* `SSH Daemon manpage <https://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8>`_
* `Ansible <https://docs.ansible.com>`_
* `Salt <https://docs.saltstack.com/>`_

---

## File: docs/deployment/hosting-repositories.rst

.. include:: ../global.rst.inc
.. highlight:: none
.. _hosting_repositories:

Hosting repositories
====================

This section shows how to provide repository storage securely for users.

Repositories are accessed through SSH. Each user of the service should
have their own login, which is only able to access that user's files.
Technically, it is possible to have multiple users share one login;
however, separating them is better. Separate logins increase isolation
and provide an additional layer of security and safety for both the
provider and the users.

For example, if a user manages to breach ``borg serve``, they can
only damage their own data (assuming that the system does not have further
vulnerabilities).

Use the standard directory structure of the operating system. Each user
is assigned a home directory, and that user's repositories reside in their
home directory.

The following ``~user/.ssh/authorized_keys`` file is the most important
piece for a correct deployment. It allows the user to log in via
their public key (which must be provided by the user), and restricts
SSH access to safe operations only.

::

  command="borg serve --restrict-to-repository /home//repository",restrict
  <key type> <key> <key host>

.. note:: The text shown above needs to be written on a **single** line!

.. warning::

    If this file should be automatically updated (e.g. by a web console),
    pay **utmost attention** to sanitizing user input. Strip all whitespace
    around the user-supplied key, ensure that it **only** contains ASCII
    with no control characters and that it consists of three parts separated
    by a single space. Ensure that no newlines are contained within the key.

The ``restrict`` keyword enables all restrictions, i.e. disables port, agent
and X11 forwarding, as well as disabling PTY allocation and execution of ~/.ssh/rc.
If any future restriction capabilities are added to authorized_keys
files they will be included in this set.

The ``command`` keyword forces execution of the specified command
upon login. This must be ``borg serve``. The ``--restrict-to-repository``
option permits access to exactly **one** repository. It can be given
multiple times to permit access to more than one repository.

The repository may not exist yet; it can be initialized by the user,
which allows for encryption.

Refer to the `sshd(8) <https://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8>`_
man page for more details on SSH options.
See also :ref:`borg_serve`

---

## File: docs/deployment/image-backup.rst

.. include:: ../global.rst.inc
.. highlight:: none

Backing up entire disk images
=============================

Backing up disk images can still be efficient with Borg because its `deduplication`_
technique makes sure only the modified parts of the file are stored. Borg also has
optional simple sparse file support for extraction.

It is of utmost importance to pin down the disk you want to back up.
Use the disk's SERIAL for that.
Use:

.. code-block:: bash

    # You can find the short disk serial by:
    # udevadm info --query=property --name=nvme1n1 | grep ID_SERIAL_SHORT | cut -d '=' -f 2
    export BORG_REPO=/path/to/repo
    DISK_SERIAL="7VS0224F"
    DISK_ID=$(readlink -f /dev/disk/by-id/*"${DISK_SERIAL}") # Returns /dev/nvme1n1

    mapfile -t PARTITIONS < <(lsblk -o NAME,TYPE -p -n -l "$DISK_ID" | awk '$2 == "part" {print $1}')
    echo "Partitions of $DISK_ID:"
    echo "${PARTITIONS[@]}"
    echo "Disk Identifier: $DISK_ID"

    # Use the following line to perform a Borg backup for the full disk:
    # borg create --read-special disk-backup "$DISK_ID"

    # Use the following to perform a Borg backup for all partitions of the disk
    # borg create --read-special partitions-backup "${PARTITIONS[@]}"

    # Example output:
    # Partitions of /dev/nvme1n1:
    # /dev/nvme1n1p1
    # /dev/nvme1n1p2
    # /dev/nvme1n1p3
    # Disk Identifier: /dev/nvme1n1
    # borg create --read-special disk-backup /dev/nvme1n1
    # borg create --read-special partitions-backup /dev/nvme1n1p1 /dev/nvme1n1p2 /dev/nvme1n1p3




Decreasing the size of image backups
------------------------------------

Disk images are as large as the full disk when uncompressed and might not get much
smaller post-deduplication after heavy use because virtually all filesystems do not
actually delete file data on disk but instead delete the filesystem entries referencing
the data. Therefore, if a disk nears capacity and files are deleted again, the change
will barely decrease the space it takes up when compressed and deduplicated. Depending
on the filesystem, there are several ways to decrease the size of a disk image:

Using ntfsclone (NTFS, i.e. Windows VMs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``ntfsclone`` can only operate on filesystems with the journal cleared (i.e. turned-off
machines), which somewhat limits its utility in the case of VM snapshots. However, when
it can be used, its special image format is even more efficient than just zeroing and
deduplicating. For backup, save the disk header and the contents of each partition::

    HEADER_SIZE=$(sfdisk -lo Start $DISK | grep -A1 -P 'Start$' | tail -n1 | xargs echo)
    PARTITIONS=$(sfdisk -lo Device,Type $DISK | sed -e '1,/Device\s*Type/d')
    dd if=$DISK count=$HEADER_SIZE | borg create --repo repo hostname-partinfo -
    echo "$PARTITIONS" | grep NTFS | cut -d' ' -f1 | while read x; do
        PARTNUM=$(echo $x | grep -Eo "[0-9]+$")
        ntfsclone -so - $x | borg create --repo repo hostname-part$PARTNUM -
    done
    # to back up non-NTFS partitions as well:
    echo "$PARTITIONS" | grep -v NTFS | cut -d' ' -f1 | while read x; do
        PARTNUM=$(echo $x | grep -Eo "[0-9]+$")
        borg create --read-special --repo repo hostname-part$PARTNUM $x
    done

Restoration is a similar process::

    borg extract --stdout --repo repo hostname-partinfo | dd of=$DISK && partprobe
    PARTITIONS=$(sfdisk -lo Device,Type $DISK | sed -e '1,/Device\s*Type/d')
    borg list --format {archive}{NL} repo | grep 'part[0-9]*$' | while read x; do
        PARTNUM=$(echo $x | grep -Eo "[0-9]+$")
        PARTITION=$(echo "$PARTITIONS" | grep -E "$DISKp?$PARTNUM" | head -n1)
        if echo "$PARTITION" | cut -d' ' -f2- | grep -q NTFS; then
            borg extract --stdout --repo repo $x | ntfsclone -rO $(echo "$PARTITION" | cut -d' ' -f1) -
        else
            borg extract --stdout --repo repo $x | dd of=$(echo "$PARTITION" | cut -d' ' -f1)
        fi
    done

.. note::

   When backing up a disk image (as opposed to a real block device), mount it as
   a loopback image to use the above snippets::

       DISK=$(losetup -Pf --show /path/to/disk/image)
       # do backup as shown above
       losetup -d $DISK

Using zerofree (ext2, ext3, ext4)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``zerofree`` works similarly to ntfsclone in that it zeros out unused chunks of the FS,
except it works in place, zeroing the original partition. This makes the backup process
a bit simpler::

    sfdisk -lo Device,Type $DISK | sed -e '1,/Device\s*Type/d' | grep Linux | cut -d' ' -f1 | xargs -n1 zerofree
    borg create --read-special --repo repo hostname-disk $DISK

Because the partitions were zeroed in place, restoration is only one command::

    borg extract --stdout --repo repo hostname-disk | dd of=$DISK

.. note:: The "traditional" way to zero out space on a partition, especially one already
          mounted, is simply to ``dd`` from ``/dev/zero`` to a temporary file and delete
          it. This is ill-advised for the reasons mentioned in the ``zerofree`` man page:

          - it is slow.
          - it makes the disk image (temporarily) grow to its maximal extent.
          - it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.

Virtual machines
----------------

If you use non-snapshotting backup tools like Borg to back up virtual machines, then
the VMs should be turned off for the duration of the backup. Backing up live VMs can
(and will) result in corrupted or inconsistent backup contents: a VM image is just a
regular file to Borg with the same issues as regular files when it comes to concurrent
reading and writing from the same file.

For backing up live VMs use filesystem snapshots on the VM host, which establishes
crash-consistency for the VM images. This means that with most filesystems (that
are journaling) the FS will always be fine in the backup (but may need a journal
replay to become accessible).

Usually this does not mean that file *contents* on the VM are consistent, since file
contents are normally not journaled. Notable exceptions are ext4 in data=journal mode,
ZFS and btrfs (unless nodatacow is used).

Applications designed with crash-consistency in mind (most relational databases like
PostgreSQL, SQLite etc. but also for example Borg repositories) should always be able
to recover to a consistent state from a backup created with crash-consistent snapshots
(even on ext4 with data=writeback or XFS). Other applications may require a lot of work
to reach application-consistency; it's a broad and complex issue that cannot be explained
in entirety here.

Hypervisor snapshots capturing most of the VM's state can also be used for backups and
can be a better alternative to pure filesystem-based snapshots of the VM's disk, since
no state is lost. Depending on the application this can be the easiest and most reliable
way to create application-consistent backups.

Borg does not intend to address these issues due to their huge complexity and
platform/software dependency. Combining Borg with the mechanisms provided by the platform
(snapshots, hypervisor features) will be the best approach to start tackling them.

---

## File: docs/deployment/non-root-user.rst

.. include:: ../global.rst.inc
.. highlight:: none
.. _non_root_user:

================================
Backing up using a non-root user
================================

This section describes how to run Borg as a non-root user and still be able to
back up every file on the system.

Normally, Borg is run as the root user to bypass all filesystem permissions and
be able to read all files. However, in theory this also allows Borg to modify or
delete files on your system (for example, in case of a bug).

To eliminate this possibility, we can run Borg as a non-root user and give it read-only
permissions to all files on the system.


Using Linux capabilities inside a systemd service
=================================================

One way to do so is to use Linux `capabilities
<https://man7.org/linux/man-pages/man7/capabilities.7.html>`_ within a systemd
service.

Linux capabilities allow us to grant parts of the root user’s privileges to
a non-root user. This works on a per-thread level and does not grant permissions
to the non-root user as a whole.

For this, we need to run the backup script from a systemd service and use the `AmbientCapabilities
<https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#AmbientCapabilities=>`_
option added in systemd 229.

A very basic unit file would look like this:

::

    [Unit]
    Description=Borg Backup

    [Service]
    Type=oneshot
    User=borg
    ExecStart=/usr/local/sbin/backup.sh

    AmbientCapabilities=CAP_DAC_READ_SEARCH

The ``CAP_DAC_READ_SEARCH`` capability gives Borg read-only access to all files and directories on the system.

This service can then be started manually using ``systemctl start``, a systemd timer or other methods.

Restore considerations
======================

Use the root user when restoring files. If you use the non-root user, ``borg extract`` will
change ownership of all restored files to the non-root user. Using ``borg mount`` will not allow the
non-root user to access files it would not be able to access on the system itself.

Other than that, you can use the same restore process you would use when running the backup as root.

.. warning::

    When using a local repository and running Borg commands as root, make sure to use only commands that do not
    modify the repository itself, such as extract or mount. Modifying the repository as root will break it for the
    non-root user, since some files inside the repository will then be owned by root.

---

## File: docs/deployment/pull-backup.rst

.. include:: ../global.rst.inc
.. highlight:: none
.. _pull_backup:

=======================
Backing up in pull mode
=======================

Typically the Borg client connects to a backup server using SSH as a transport
when initiating a backup. This is referred to as push mode.

However, if you require the backup server to initiate the connection, or prefer
it to initiate the backup run, one of the following workarounds is required to
allow such a pull-mode setup.

A common use case for pull mode is to back up a remote server to a local personal
computer.

SSHFS
=====

Assume you have a pull backup system set up with Borg, where a backup server
pulls data from the target via SSHFS. In this mode, the backup client's filesystem
is mounted remotely on the backup server. Pull mode is even possible if
the SSH connection must be established by the client via a remote tunnel. Other
network file systems like NFS or SMB could be used as well, but SSHFS is very
simple to set up and probably the most secure one.

There are some restrictions caused by SSHFS. For example, unless you define UID
and GID mappings when mounting via ``sshfs``, owners and groups of the mounted
filesystem will probably change, and you may not have access to those files if
Borg is not run with root privileges.

SSHFS is a FUSE filesystem and uses the SFTP protocol, so there may also be
unsupported features that the actual implementations of SSHFS, libfuse, and
SFTP on the backup server do not support, like filename encodings, ACLs, xattrs,
or flags. Therefore, there is no guarantee that you can restore a system
completely in every aspect from such a backup.

.. warning::

    To mount the client's root filesystem you will need root access to the
    client. This contradicts the usual threat model of Borg, where
    clients do not need to trust the backup server (data is encrypted). In pull
    mode the server (when logged in as root) could cause unlimited damage to the
    client. Therefore, pull mode should be used only with servers you fully
    trust!

.. warning::

    Additionally, while chrooted into the client's root filesystem,
    code from the client will be executed. Therefore, you should do this only when
    you fully trust the client.

.. warning::

    The SSHFS + chroot method requires that the backup server and the client
    use the same CPU architecture (e.g. both x86_64), as binaries from the
    client (like ``/bin/sh`` or the ``borg`` binary) are executed on the
    backup server. If you have different architectures, use the :ref:`socat_method`
    instead.

.. warning::

    The chroot method was chosen to get the right user and group name-id
    mappings, assuming they only come from files (/etc/passwd and /etc/group).
    This assumption might be wrong, e.g. if users/groups also come from
    ldap or other providers.
    Thus, it might be better to use ``--numeric-ids`` and not archive any
    user or group names (but just the numeric IDs) and not use chroot.

Creating a backup
-----------------

Generally, in a pull backup situation there is no direct way for borg to know
the client's original UID:GID name mapping of files, because Borg would use
``/etc/passwd`` and ``/etc/group`` of the backup server to map the names. To
derive the right names, Borg needs to have access to the client's passwd and
group files and use them in the backup process.

The solution to this problem is chrooting into an sshfs mounted directory. In
this example the whole client root file system is mounted. We use the
stand-alone BorgBackup executable and copy it into the mounted file system to
make Borg available after entering chroot; this can be skipped if Borg is
already installed on the client.

::

    # Mount client root file system.
    mkdir /tmp/sshfs
    sshfs root@host:/ /tmp/sshfs
    # Mount BorgBackup repository inside it.
    mkdir /tmp/sshfs/borgrepo
    mount --bind /path/to/repo /tmp/sshfs/borgrepo
    # Make borg executable available.
    cp /usr/local/bin/borg /tmp/sshfs/usr/local/bin/borg
    # Mount important system directories and enter chroot.
    cd /tmp/sshfs
    for i in dev proc sys; do mount --bind /$i $i; done
    chroot /tmp/sshfs

Now we are on the backup system but inside a chroot with the client's root file
system. We have a copy of Borg binary in ``/usr/local/bin`` and the repository
in ``/borgrepo``. Borg will back up the client's user/group names, and we can
create the backup, retaining the original paths, excluding the repository:

::

    borg create --exclude borgrepo --files-cache ctime,size --repo /borgrepo archive /

For the sake of simplicity only ``borgrepo`` is excluded here. You may want to
set up an exclude file with additional files and folders to be excluded. Also
note that we have to modify Borg's file change detection behaviour – SSHFS
cannot guarantee stable inode numbers, so we have to supply the
``--files-cache`` option.

Finally, we need to exit chroot, unmount all the stuff and clean up:

::

    exit # exit chroot
    rm /tmp/sshfs/usr/local/bin/borg
    cd /tmp/sshfs
    for i in dev proc sys borgrepo; do umount ./$i; done
    rmdir borgrepo
    cd ~
    umount /tmp/sshfs
    rmdir /tmp/sshfs

Thanks to secuser on IRC for this how-to!

Restore methods
---------------

The counterpart of a pull backup is a push restore. Depending on the type of
restore – full restore or partial restore – there are different methods to make
sure the correct IDs are restored.

Partial restore

# Mount client root file system.
mkdir /tmp/sshfs
sshfs root@host:/ /tmp/sshfs
# Mount BorgBackup repository inside it.
mkdir /tmp/sshfs/borgrepo
mount --bind /path/to/repo /tmp/sshfs/borgrepo
# Make borg executable available.
cp /usr/local/bin/borg /tmp/sshfs/usr/local/bin/borg
# Mount important system directories and enter chroot.
cd /tmp/sshfs
for i in dev proc sys; do mount --bind /$i $i; done
chroot /tmp/sshfs

borg extract --repo /borgrepo archive PATH

exit # exit chroot
rm /tmp/sshfs/usr/local/bin/borg
cd /tmp/sshfs
for i in dev proc sys borgrepo; do umount ./$i; done
rmdir borgrepo
cd ~
umount /tmp/sshfs
rmdir /tmp/sshfs

When doing a full restore, we restore all files (including the ones containing
the ID-to-name mapping, ``/etc/passwd`` and ``/etc/group``). Everything will be
consistent automatically if we restore the numeric IDs stored in the archive. So
there is no need for a chroot environment; we just mount the client file system
and extract a backup, utilizing the ``--numeric-ids`` option:

::

    sshfs root@host:/ /mnt/sshfs
    cd /mnt/sshfs
    borg extract --numeric-ids --repo /path/to/repo archive
    cd ~
    umount /mnt/sshfs

Simple (lossy) full restore

borg export-tar --repo /path/to/repo archive - | ssh root@host 'tar -C / -x'

Warning: remote port forwarding failed for listen path /run/borg/reponame.sock

Remote: YYYY/MM/DD HH:MM:SS socat[XXX] E connect(5, AF=1 "/run/borg/reponame.sock", 13): Connection refused
  Connection closed by remote host. Is borg working on the server?

Do this once for each client on *borg-server* to allow *borgs* to connect itself on *borg-server* using a
dedicated ssh key:

::

  borgs@borg-server$ install -m 700 -d ~/.ssh/
  borgs@borg-server$ ssh-keygen -N '' -t rsa -f ~/.ssh/borg-client_key
  borgs@borg-server$ { echo -n 'command="borg serve --restrict-to-repo ~/repo",restrict '; cat ~/.ssh/borg-client_key.pub; } >> ~/.ssh/authorized_keys
  borgs@borg-server$ chmod 600 ~/.ssh/authorized_keys

``install -m 700 -d ~/.ssh/``

  Create directory ~/.ssh with correct permissions if it does not exist yet.

``ssh-keygen -N '' -t rsa -f ~/.ssh/borg-client_key``

  Create an ssh key dedicated to communication with borg-client.

.. note::
  Another more complex approach is using a unique ssh key for each pull operation.
  This is more secure as it guarantees that the key will not be used for other purposes.

``{ echo -n 'command="borg serve --restrict-to-repo ~/repo",restrict '; cat ~/.ssh/borg-client_key.pub; } >> ~/.ssh/authorized_keys``

  Add borg-client's ssh public key to ~/.ssh/authorized_keys with forced command and restricted mode.
  The borg client is restricted to use one repo at the specified path.

``chmod 600 ~/.ssh/authorized_keys``

  Fix permissions of ~/.ssh/authorized_keys.

Pull operation
~~~~~~~~~~~~~~

Initiating borg command execution from *borg-server* (e.g. init)::

  borgs@borg-server$ (
    eval $(ssh-agent) > /dev/null
    ssh-add -q ~/.ssh/borg-client_key
    echo 'your secure borg key passphrase' | \
      ssh -A -o StrictHostKeyChecking=no borgc@borg-client "BORG_PASSPHRASE=\$(cat) BORG_RSH='ssh -o StrictHostKeyChecking=no' borg init --encryption repokey ssh://borgs@borg-server/~/repo"
    kill "${SSH_AGENT_PID}"
  )

Parentheses around commands are needed to avoid interference with a possibly already running ssh-agent.
Parentheses are not needed when using a dedicated bash process.

``eval $(ssh-agent) > /dev/null``

  Run the SSH agent in the background and export related environment variables to the current bash session.

``ssh-add -q ~/.ssh/borg-client_key``

  Load the SSH private key dedicated to communication with the borg-client into the SSH agent.
  Look at ``man 1 ssh-add`` for a more detailed explanation.

.. note::
  Care needs to be taken when loading keys into the SSH agent. Users on the *borg-client* having read/write permissions
  to the agent's UNIX-domain socket (at least borgc and root in our case) can access the agent on *borg-server* through
  the forwarded connection and can authenticate using any of the identities loaded into the agent
  (look at ``man 1 ssh`` for more detailed explanation). Therefore there are some security considerations:

  * Private keys loaded into the agent must not be used to enable access anywhere else.
  * The keys meant to be loaded into the agent must be specified explicitly, not from default locations.
  * The *borg-client*'s entry in *borgs@borg-server:~/.ssh/authorized_keys* must be as restrictive as possible.

``echo 'your secure borg key passphrase' | ssh -A -o StrictHostKeyChecking=no borgc@borg-client "BORG_PASSPHRASE=\$(cat) BORG_RSH='ssh -o StrictHostKeyChecking=no' borg init --encryption repokey ssh://borgs@borg-server/~/repo"``

  Run the *borg init* command on *borg-client*.

  *ssh://borgs@borg-server/~/repo* refers to the repository *repo* within borgs's home directory on *borg-server*.

  *StrictHostKeyChecking=no* is used to add host keys automatically to *~/.ssh/known_hosts* without user intervention.

``kill "${SSH_AGENT_PID}"``

  Kill ssh-agent with loaded keys when it is not needed anymore.

Remote forwarding
=================

The standard ssh client allows to create tunnels to forward local ports to a remote server (local forwarding) and also
to allow remote ports to be forwarded to local ports (remote forwarding).

This remote forwarding can be used to allow remote backup clients to access the backup server even if the backup server
cannot be reached by the backup client.

This can even be used in cases where neither the backup server can reach the backup client and the backup client cannot
reach the backup server, but some intermediate host can access both.

A schematic approach is as follows

::

      Backup Server (backup@mybackup)          Intermediate Machine (john@myinter)              Backup Client (bob@myclient)

                                              1. Establish SSH remote forwarding  ----------->  SSH listen on local port

                                                                                                2. Starting ``borg create`` establishes
                                              3. SSH forwards to intermediate machine  <------- SSH connection to the local port
      4. Receives backup connection <-------  and further on to backup server
      via SSH

So for the backup client the backup is done via SSH to a local port and for the backup server there is a normal backup
performed via ssh.

In order to achieve this, the following commands can be used to create the remote port forwarding:

1. On machine ``myinter``

``ssh bob@myclient -v -C -R 8022:mybackup:22 -N``

This will listen for ssh-connections on port ``8022`` on ``myclient`` and forward connections to port 22 on ``mybackup``.

You can also remove the need for machine ``myinter`` and create the port forwarding on the backup server directly by
using ``localhost`` instead of ``mybackup``

2. On machine ``myclient``

``borg create -v --progress --stats ssh://backup@localhost:8022/home/backup/repos/myclient /``

Make sure to use port ``8022`` and ``localhost`` for the repository as this instructs borg on ``myclient`` to use the
remote forwarded ssh connection.

SSH Keys
--------

If you want to automate backups when using this method, the ssh ``known_hosts`` and ``authorized_keys`` need to be set up
to allow connections.

Security Considerations
-----------------------

Opening up SSH access this way can pose a security risk as it effectively opens remote access to your
backup server on the client even if it is located outside of your company network.

To reduce the chances of compromise, you should configure a forced command in ``authorized_keys`` to prevent
anyone from performing any other action on the backup server.

This can be done e.g. by adding the following in ``$HOME/.ssh/authorized_keys`` on ``mybackup`` with proper
path and client-fqdn:

::

  command="cd /home/backup/repos/<client fqdn>;borg serve --restrict-to-path /home/backup/repos/<client fqdn>"


All the additional security considerations for borg should be applied, see :ref:`central-backup-server` for some additional
hints.

More information
----------------

See `remote forwarding`_ and the `ssh man page`_ for more information about remote forwarding.

   .. _remote forwarding: https://linuxize.com/post/how-to-setup-ssh-tunneling/
   .. _ssh man page: https://manpages.debian.org/testing/manpages-de/ssh.1.de.html

---

## File: docs/internals/data-structures.rst

.. include:: ../global.rst.inc
.. highlight:: none

.. _data-structures:

Data structures and file formats
================================

This page documents the internal data structures and storage
mechanisms of Borg. It is partly based on mailing list
discussions and also on static code analysis.

.. todo:: Clarify terms, perhaps create a glossary.
          ID (client?) vs. key (repository?),
          chunks (blob of data in repo?) vs. object (blob of data in repo, referred to from another object?),

.. _repository:

Repository
----------

Borg stores its data in a `Repository`, which is a key-value store and has
the following structure:

config/
  readme
    simple text object telling that this is a Borg repository
  id
    the unique repository ID encoded as hexadecimal number text
  version
    the repository version encoded as decimal number text
  manifest
    some data about the repository, binary
  space-reserve.N
    purely random binary data to reserve space, e.g. for disk-full emergencies

cache/
  checked-packs
    repository check results (pack id -> timestamp, result), as a hashtable with an
    appended integrity hash. Records are kept across checks: ``check --max-age``
    skips packs whose intact record is younger than the given age, and partial checks
    (``--max-duration``) verify the least-recently-checked packs first so repeated
    runs cover the whole repository. Records of corrupt packs are kept for repair and
    always re-verified. Records of packs no longer listed in packs/ are pruned when a
    check finishes.

There is a list of pointers to archive objects in this directory:

archives/
  0000... .. ffff...

The actual data is stored into a nested directory structure, using the full
object ID as name. Each (encrypted and compressed) object is stored separately.

data/
  00/ .. ff/
    00/ .. ff/
      0000... .. ffff...

keys/
    When using repokey mode, the encrypted, passphrase protected borg keys are
    stored here as a base64 encoded text. The sha256 content hash of the
    stored borg key is used for the name.

    A repository may contain *multiple* such borg keys (one per passphrase) to
    support the :ref:`multiple borg keys ` feature.
    keyfile and repokey borg keys use the same format and naming (only the
    storage location differs).

locks/
  used by the locking system to manage shared and exclusive locks.


Keys
~~~~

Repository object IDs (which are used as key into the key-value store) are
byte strings of fixed length (256-bit, 32 bytes), computed like this::

  key = id = id_hash(plaintext_data)  # plain = not encrypted, not compressed, not obfuscated

The id_hash function is selected via ``borg repo-create --id-hash`` (independently
of ``--encryption``). For encrypted repositories it is a keyed MAC over the
plaintext (keyed by ``id_key``): ``sha256`` selects HMAC-SHA256, ``blake3``
selects a keyed BLAKE3. The unencrypted ``none`` mode uses a plain ``sha256``.

As the id / key is used for deduplication, id_hash must be a cryptographically
strong hash or MAC.

Repository objects
~~~~~~~~~~~~~~~~~~

Each repository object is stored separately, under its ID into data/xx/yy/xxyy...

A repo object has a structure like this:

* 32-bit meta size
* 32-bit data size
* meta
* data

The overall size of repository objects varies from very small (a small source
file will be stored as a single repository object) to medium (big source files will
be cut into medium-sized chunks of some MB).

Metadata and data are separately encrypted and authenticated (depending on
the user's choices).

See :ref:`data-encryption` for a graphic outlining the anatomy of the
encryption.

Repo object metadata
~~~~~~~~~~~~~~~~~~~~

Metadata is a MessagePack-encoded (and encrypted/authenticated) dict with:

- ctype (compression type 0..255)
- clevel (compression level, one byte, interpreted depending on ctype - see
  :ref:`data-compression`)
- csize (overall compressed (and maybe obfuscated) data size)
- psize (only when obfuscated: payload size without the obfuscation trailer)
- size (uncompressed size of the data)

Having this separately encrypted metadata makes it more efficient to query
the metadata without having to read, transfer and decrypt the (usually much
bigger) data part.

The compression `ctype` and `clevel` is explained in :ref:`data-compression`.


Compaction
~~~~~~~~~~

``borg compact`` is used to free repository space. It will:

- list all object IDs present in the repository
- read all archives and determine which object IDs are in use
- remove all unused objects from the repository
- inform / warn about anything remarkable it found:

  - warn about IDs used, but not present (data loss!)
  - inform about IDs that reappeared that were previously lost
- compute statistics about:

  - compression and deduplication factors
  - repository space usage and space freed


The object graph
----------------

On top of the simple key-value store offered by the Repository_,
Borg builds a much more sophisticated data structure that is essentially
a completely encrypted object graph. Objects, such as archives_, are referenced
by their chunk ID, which is cryptographically derived from their contents.
More on how this helps security in :ref:`security_structural_auth`.

.. figure:: object-graph.png
    :figwidth: 100%
    :width: 100%

.. _manifest:

The manifest
~~~~~~~~~~~~

Compared to borg 1.x:

- the manifest moved from object ID 0 to config/manifest
- the archives list has been moved from the manifest to archives/*

The manifest is rewritten each time an archive is created, deleted,
or modified. It looks like this:

.. code-block:: python

    {
        'version': 1,
        'timestamp': '2017-05-05T12:42:23.042864',
        'item_keys': ['acl_access', 'acl_default', ...],
        'config': {},
        'archives': {
            '2017-05-05-system-backup': {
                'id': b'<32 byte binary object ID>',
                'time': '2017-05-05T12:42:22.942864',
            },
        },
    }

The *version* field can be either 1 or 2. The versions differ in the
way feature flags are handled, described below.

The *timestamp* field is used to avoid logical replay attacks where
the server just resets the repository to a previous state.

*item_keys* is a list containing all Item_ keys that may be encountered in
the repository. It is used by *borg check*, which verifies that all keys
in all items are a subset of these keys. Thus, an older version of *borg check*
supporting this mechanism can correctly detect keys introduced in later versions.

*config* is a general-purpose location for additional metadata. All versions
of Borg preserve its contents.

Feature flags
+++++++++++++

Feature flags are used to add features to data structures without causing
corruption if older versions are used to access or modify them. The main issues
to consider for a feature flag oriented design are flag granularity,
flag storage, and cache_ invalidation.

Feature flags are divided in approximately three categories, detailed below.
Due to the nature of ID-based deduplication, write (i.e. creating archives) and
read access are not symmetric; it is possible to create archives referencing
chunks that are not readable with the current feature set. The third
category are operations that require accurate reference counts, for example
archive deletion and check.

As the manifest is always updated and always read, it is the ideal place to store
feature flags, comparable to the super-block of a file system. The only problem
is to recover from a lost manifest, i.e. how is it possible to detect which feature
flags are enabled, if there is no manifest to tell. This issue is left open at this time,
but is not expected to be a major hurdle; it doesn't have to be handled efficiently, it just
needs to be handled.

Lastly, cache_ invalidation is handled by noting which feature
flags were and which were not understood while manipulating a cache.
This allows borg to detect whether the cache needs to be invalidated,
i.e. rebuilt from scratch. See `Cache feature flags`_ below.

The *config* key stores the feature flags enabled on a repository:

.. code-block:: python

    config = {
        'feature_flags': {
            'read': {
                'mandatory': ['some_feature'],
            },
            'check': {
                'mandatory': ['other_feature'],
            }
            'write': ...,
            'delete': ...
        },
    }

The top-level distinction for feature flags is the operation the client intends
to perform,

| the *read* operation includes extraction and listing of archives,
| the *write* operation includes creating new archives,
| the *delete* (archives) operation,
| the *check* operation requires full understanding of everything in the repository.
|

These are weakly set-ordered; *check* will include everything required for *delete*,
*delete* will likely include *write* and *read*. However, *read* may require more
features than *write* (due to ID-based deduplication, *write* does not necessarily
require reading/understanding repository contents).

Each operation can contain several sets of feature flags. Only one set,
the *mandatory* set is currently defined.

Upon reading the manifest, the Borg client has already determined which operation
should be performed. If feature flags are found in the manifest, the set
of feature flags supported by the client is compared to the mandatory set
found in the manifest. If any unsupported flags are found (i.e. the mandatory set is
not a subset of the features supported by the Borg client used), the operation
is aborted with a *MandatoryFeatureUnsupported* error:

    Unsupported repository feature(s) {'some_feature'}. A newer version of borg is required to access this repository.

Older Borg releases do not have this concept and do not perform feature flags checks.
These can be locked out with manifest version 2. Thus, the only difference between
manifest versions 1 and 2 is that the latter is only accepted by Borg releases
implementing feature flags.

Therefore, as soon as any mandatory feature flag is enabled in a repository,
the manifest version must be switched to version 2 in order to lock out all
Borg releases unaware of feature flags.

.. _Cache feature flags:
.. rubric:: Cache feature flags

`The cache <cache>`_ does not have its separate set of feature flags. Instead, Borg stores
which flags were used to create or modify a cache.

All mandatory manifest features from all operations are gathered in one set.
Then, two sets of features are computed;

- those features that are supported by the client and mandated by the manifest
  are added to the *mandatory_features* set,
- the *ignored_features* set comprised of those features mandated by the manifest,
  but not supported by the client.

Because the client previously checked compliance with the mandatory set of features
required for the particular operation it is executing, the *mandatory_features* set
will contain all necessary features required for using the cache safely.

Conversely, the *ignored_features* set contains only those features which were not
relevant to operating the cache. Otherwise, the client would not pass the feature
set test against the manifest.

When opening a cache and the *mandatory_features* set is not a subset of the features
supported by the client, the cache is wiped out and rebuilt,
since a client not supporting a mandatory feature that the cache was built with
would be unable to update it correctly.
The assumption behind this behaviour is that any of the unsupported features could have
been reflected in the cache and there is no way for the client to discern whether
that is the case.
Meanwhile, it may not be practical for every feature to have clients using it track
whether the feature had an impact on the cache.
Therefore, the cache is wiped.

When opening a cache and the intersection of *ignored_features* and the features
supported by the client contains any elements, i.e. the client possesses features
that the previous client did not have and those new features are enabled in the repository,
the cache is wiped out and rebuilt.

While the former condition likely requires no tweaks, the latter condition is formulated
in an especially conservative way to play it safe. It seems likely that specific features
might be exempted from the latter condition.

.. rubric:: Defined feature flags

Currently no feature flags are defined.

From currently planned features, some examples follow,
these may/may not be implemented and purely serve as examples.

- A mandatory *read* feature could be using a different encryption scheme (e.g. session keys).
  This may not be mandatory for the *write* operation - reading data is not strictly required for
  creating an archive.
- Any additions to the way chunks are referenced (e.g. to support larger archives) would
  become a mandatory *delete* and *check* feature; *delete* implies knowing correct
  reference counts, so all object references need to be understood. *check* must
  discover the entire object graph as well, otherwise the "orphan chunks check"
  could delete data still in use.

.. _archive:

Archives
~~~~~~~~

Each archive is an object referenced by an entry below archives/.
The archive object itself does not store any of the data contained in the
archive it describes.

Instead, it contains a list of chunks which form a msgpacked stream of items_.
The archive object itself further contains some metadata:

* *version*
* *name*, which might differ from the name set in the archives/* object.
  When :ref:`borg_check` rebuilds the manifest (e.g. if it was corrupted) and finds
  more than one archive object with the same name, it adds a counter to the name
  in archives/*, but leaves the *name* field of the archives as they were.
* *item_ptrs*, a list of "pointer chunk" IDs.
  Each "pointer chunk" contains a list of chunk IDs of item metadata.
* *command_line*, the command line which was used to create the archive
* *hostname*
* *username*
* *time* and *time_end* are the start and end timestamps, respectively
* *comment*, a user-specified archive comment
* *chunker_params* are the :ref:`chunker-params <chunker-params>` used for creating the archive.
  This is used by :ref:`borg_recreate` to determine whether a given archive needs rechunking.
* Some other pieces of information related to recreate.

.. _item:

Items
~~~~~

Each item represents a file, directory or other file system item and is stored as a
dictionary created by the ``Item`` class that contains:

* path
* list of data chunks (size: count * ~40B)
* user
* group
* uid
* gid
* mode (item type + permissions)
* source (for symlinks)
* hlid (for hardlinks)
* rdev (for device files)
* mtime, atime, ctime, birthtime in nanoseconds
* xattrs
* acl (various OS-dependent fields)
* flags

All items are serialized using msgpack and the resulting byte stream
is fed into the same chunker algorithm as used for regular file data
and turned into deduplicated chunks. The reference to these chunks is then added
to the archive metadata. To achieve a finer granularity on this metadata
stream, we use different chunker params for this chunker, which result in
smaller chunks.

A chunk is stored as an object as well, of course.

.. _chunks:
.. _chunker_details:

Chunks
~~~~~~

Borg has these chunkers (the default is "fastcdc"):

- "fixed": a simple, low cpu overhead, fixed blocksize chunker, optionally
  supporting a header block of different size.
- "fastcdc": variable, content-defined blocksize, uses the window-less, keyed
  Gear rolling hash (FastCDC_); faster than buzhash, same deduplication.
- "buzhash64": similar to "buzhash", but improved 64bit implementation
- "buzhash": variable, content-defined blocksize, uses a rolling hash
  computed by the Buzhash_ algorithm.
- "toeplitz-aes": like "rabin-aes", but the universal hash is a tabulated
  LFSR/Toeplitz hash (secret 2 KiB table, fixed public polynomial); same
  speed as "rabin-aes" with the best collision bound of the three.
- "rabin-aes": variable, content-defined blocksize; a rolling Rabin fingerprint
  (secret polynomial) post-processed with AES-128, so the cut decision only
  depends on the AES output ("UHF-then-PRF" construction). Strongest available
  protection against chunk-size fingerprinting attacks.
- "goldilocks-aes": like "rabin-aes", but the universal hash is a polynomial
  hash over the Goldilocks prime field (the reference construction of the
  underlying paper); about half the rabin-aes speed, mainly a comparison
  baseline.

All chunkers support sparse file processing (``borg create --sparse``): hole
ranges in the input file are then detected (via ``SEEK_HOLE``/``SEEK_DATA``)
and seeked over instead of being read, processing their content as all-zero.

For some more general usage hints see also ``--chunker-params``.

"fixed" chunker
+++++++++++++++

The fixed chunker triggers (chunks) at even-spaced offsets, e.g. every 4MiB,
producing chunks of same block size (the last chunk is not required to be
full-size).

Optionally, it supports processing a differently sized "header" first, before
it starts to cut chunks of the desired block size.
The default is not to have a differently sized header.

``borg create --chunker-params fixed,BLOCK_SIZE[,HEADER_SIZE]``

- BLOCK_SIZE: no default value, multiple of the system page size (usually 4096
  bytes) recommended. E.g.: 4194304 would cut 4MiB sized chunks.
- HEADER_SIZE: optional, defaults to 0 (no header).

"fastcdc" chunker
+++++++++++++++++

FastCDC_ content-defined chunker using the Gear rolling hash. Unlike buzhash it
is window-less (each byte's influence simply decays out of the hash), so its
update is cheaper and it chunks noticeably faster, while producing the same
deduplication and (with normalized chunking) the same chunk-size distribution.

Like "buzhash64", the Gear table is cryptographically derived from secret key
material, so chunk cut points are unpredictable without the key.

``borg create --chunker-params fastcdc,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,NC_LEVEL``
can be used to tune the chunker parameters, the default is:

- CHUNK_MIN_EXP = 19 (minimum chunk size = 2^19 B = 512 kiB)
- CHUNK_MAX_EXP = 23 (maximum chunk size = 2^23 B = 8 MiB)
- HASH_MASK_BITS = 21 (target chunk size ~= 2^21 B = 2 MiB)
- NC_LEVEL = 2 (normalized chunking level, 0 disables it)

There is no window size (Gear is window-less). Normalized chunking varies the
cut-point mask around the target size, which tightens the chunk-size
distribution and reduces clamping at the min./max. chunk size.

This is the default chunker (``fastcdc,19,23,21,2``), also used for the item
metadata stream (with a finer granularity, ``fastcdc,15,19,17,2``).

"buzhash64" chunker
+++++++++++++++++++

Similar to "buzhash", but using 64bit wide hash values.

The buzhash table is cryptographically derived from secret key material.

These changes should improve resistance against attacks and also solve
some of the issues of the original (32bit / XORed table) implementation.

"buzhash" chunker
+++++++++++++++++

The buzhash chunker triggers (chunks) when the last HASH_MASK_BITS bits of the
hash are zero, producing chunks with a target size of 2^HASH_MASK_BITS bytes.

Buzhash is **only** used for cutting the chunks at places defined by the
content, the buzhash value is **not** used as the deduplication criteria (we
use a cryptographically strong hash/MAC over the chunk contents for this, the
id_hash).

The idea of content-defined chunking is assigning every byte where a
cut *could* be placed a hash. The hash is based on some number of bytes
(the window size) before the byte in question. Chunks are cut
where the hash satisfies some condition
(usually "n numbers of trailing/leading zeroes"). This causes chunks to be cut
in the same location relative to the file's contents, even if bytes are inserted
or removed before/after a cut, as long as the bytes within the window stay the same.
This results in a high chance that a single cluster of changes to a file will only
result in 1-2 new chunks, aiding deduplication.

Using normal hash functions this would be extremely slow,
requiring hashing approximately ``window size * file size`` bytes.
A rolling hash is used instead, which allows to add a new input byte and
compute a new hash as well as *remove* a previously added input byte
from the computed hash. This makes the cost of computing a hash for each
input byte largely independent of the window size.

Borg defines minimum and maximum chunk sizes (CHUNK_MIN_EXP and CHUNK_MAX_EXP, respectively)
which narrows down where cuts may be made, greatly reducing the amount of data
that is actually hashed for content-defined chunking.

``borg create --chunker-params buzhash,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,HASH_WINDOW_SIZE``
can be used to tune the chunker parameters, the usual values are:

- CHUNK_MIN_EXP = 19 (minimum chunk size = 2^19 B = 512 kiB)
- CHUNK_MAX_EXP = 23 (maximum chunk size = 2^23 B = 8 MiB)
- HASH_MASK_BITS = 21 (target chunk size ~= 2^21 B = 2 MiB)
- HASH_WINDOW_SIZE = 4095 [B] (`0xFFF`) (must be an odd number)

The buzhash table is altered by XORing it with a seed randomly generated once
for the repository, and stored encrypted in the keyfile. This is to prevent
chunk size based fingerprinting attacks on your encrypted repo contents (to
guess what files you have based on a specific set of chunk sizes).

"toeplitz-aes" chunker
++++++++++++++++++++++

Like "rabin-aes", but the universal hash is a tabulated LFSR-based Toeplitz
hash (Krawczyk, CRYPTO '94): the digest of the 64-byte window is
sum_j x^(63-j) * T[b_j] over GF(2)[x] mod P, where T is a secret random
table of 256 64-bit values (2 KiB of key material) and P is a *fixed public*
irreducible polynomial of degree 64. The AES-128 PRF layer is the same as
for "rabin-aes". Two distinct windows collide with probability exactly
2^-64 over the choice of T - the best possible bound for a 64-bit digest,
and unconditional (no secret polynomial sampling). The rolling update
contains no secret-dependent memory access. Speed is on par with
"rabin-aes". See :doc:`chunker` for a comparison of all chunkers.

``borg create --chunker-params toeplitz-aes,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,NC_LEVEL``

The window size is fixed at 64 bytes. NC_LEVEL is the normalized chunking
level (0 disables it); 2 is a good default. E.g.: ``toeplitz-aes,19,23,21,2``.

"rabin-aes" chunker
+++++++++++++++++++

A "UHF-then-PRF" content-defined chunker, following the provably secure
construction of `Breaking and Fixing Content-Defined Chunking
<https://eprint.iacr.org/2025/558>`_ (Truong et al., 2025): a rolling Rabin
fingerprint over GF(2)[x]/P(x) - with P a secret, random, irreducible
polynomial of degree 64 - compresses the last 64 bytes into a digest
(a universal hash), and AES-128 with a secret key is applied to that digest.
The cut decision only looks at the AES output, so observed chunk boundaries
are pseudo-random and do not provide usable equations about the chunking
secrets, unlike chunkers that cut directly on (keyed) rolling hash bits.
Both secrets are derived from the repository key material.

This is the recommended chunker when resistance against chunk-size
fingerprinting attacks matters most. It is slower than "fastcdc" (one AES
block encryption per scanned byte), but still fast in absolute terms: the
implementation batches the AES work through OpenSSL or uses AES hardware
instructions (arm64 crypto extensions / x86-64 AES-NI) where available.

``borg create --chunker-params rabin-aes,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,NC_LEVEL``

The window size is fixed at 64 bytes. NC_LEVEL is the normalized chunking
level (0 disables it); 2 is a good default. E.g.: ``rabin-aes,19,23,21,2``.

"goldilocks-aes" chunker
++++++++++++++++++++++++

Like "rabin-aes", but the universal hash is the reference construction of the
same paper: a polynomial hash over the Goldilocks prime field GF(p) with
p = 2^64 - 2^32 + 1, evaluated at a secret random point K over the same
64-byte window. The AES-128 PRF layer and the security properties are the
same as for "rabin-aes" (the two-window collision bound is even slightly
better). It is about half as fast as "rabin-aes" - prime-field multiplies
instead of table lookups in the rolling hash - and is provided mainly as a
well-understood comparison baseline.

``borg create --chunker-params goldilocks-aes,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,NC_LEVEL``

The window size is fixed at 64 bytes. NC_LEVEL is the normalized chunking
level (0 disables it); 2 is a good default. E.g.: ``goldilocks-aes,19,23,21,2``.

.. _cache:

The files cache
---------------

The **files cache** is stored in ``cache/files.<SUFFIX>`` and is used at backup
time to quickly determine whether a given file is unchanged and we have all its
chunks.

In memory, the files cache is a key -> value mapping (a Python *dict*) and contains:

* key: id_hash of the encoded path (same path as seen in archive)
* value:

  - age (0 [newest], ..., BORG_FILES_CACHE_TTL - 1)
  - file inode number
  - file size
  - file ctime_ns
  - file mtime_ns
  - list of chunk (id, size) tuples representing the file's contents

To determine whether a file has not changed, cached values are looked up via
the key in the mapping and compared to the current file attribute values.

If the file's size, timestamp and inode number is still the same, it is
considered not to have changed. In that case, we check that all file content
chunks are (still) present in the repository (we check that via the chunks
cache).

If everything is matching and all chunks are present, the file is not read /
chunked / hashed again (but still a file metadata item is written to the
archive, made from fresh file metadata read from the filesystem). This is
what makes borg so fast when processing unchanged files.

If there is a mismatch or a chunk is missing, the file is read / chunked /
hashed. Chunks already present in repo won't be transferred to repo again.

The inode number is stored and compared to make sure we distinguish between
different files, as a single path may not be unique across different
archives in different setups.

Not all filesystems have stable inode numbers. If that is the case, borg can
be told to ignore the inode number in the check via --files-cache.

The age value is used for cache management. If a file is "seen" in a backup
run, its age is reset to 0, otherwise its age is incremented by one.
If a file was not seen in BORG_FILES_CACHE_TTL backups, its cache entry is
removed.

The files cache is a python dictionary. To keep the memory overhead of python
objects low, the value is not kept as a python tuple, but in a "compressed" form:

- the chunks list is reduced from (256bit chunk id, 32bit size) tuples to bare
  32bit indexes into the chunks index (see ``ChunkIndex.k_to_idx``). The chunk
  id and size are looked up from the chunks index again when the entry is used.
  This only works while that chunks index is in memory.
- the resulting entry is then msgpacked, so one dict value is a single ``bytes``
  object instead of a nested structure of python objects.

Borg can also work without using the files cache (saves memory if you have a
lot of files or not much RAM free), then all files are assumed to have changed.
This is usually much slower than with files cache.

The on-disk format of the files cache is a stream of msgpacked tuples (key, value).
There, the chunks list is stored in its uncompressed form (chunk id and size), as
the chunks index indexes are only valid for one specific in-memory chunks index.
Loading the files cache involves reading the file, one msgpack object at a time,
unpacking it, and compressing the entry as described above.

.. _index:

The chunks index
----------------

The **chunks index** is persisted in the repository as index fragments and loaded in memory.
It is used to determine whether we already have a specific chunk.

The chunks index is a key -> value mapping and contains:

* key (32 bytes):

  - chunk id_hash
* value (48 bytes, ``ChunkIndexEntry`` in ``borg.hashindex``):

  - flags (32bit): ``F_USED`` (chunk is used / referenced), ``F_COMPRESS`` (chunk
    shall get re-compressed), ``F_PENDING`` (the chunk is still buffered in the pack
    writer, so its pack location is not resolved yet). The upper 8 bits are reserved
    for system flags (currently ``F_NEW``) and are not visible to users of the index.
  - size (32bit): plaintext chunk size, 0 if not known (see below)
  - pack_id (32 bytes): id of the pack file the chunk's blob is stored in
  - obj_offset (32bit): byte offset of the blob inside that pack file
  - obj_size (32bit): blob length (header + encrypted_meta + encrypted_data)

The last 3 values are the chunk's location, see :ref:`pack-index-entry`: reading a
chunk is one ranged read of ``[obj_offset, obj_offset + obj_size)`` from
``packs/<hex(pack_id)>``.

So a chunks index entry is 32 + 48 == 80 bytes, and that is also exactly what it
needs on disk (the serialized format is just key/value pairs, no padding, plus a
small header). In memory, there is some additional overhead, see below.

Not all of that is persisted, though: when an index fragment is written, flags and
size are zeroed (only the chunk id and the pack location are of interest there).
Thus, a chunks index that was just built from the repository has size == 0 for all
its entries, no matter whether it came from the index fragments or from the slow
rebuild (which reads the pack headers, where only the stored blob size is known,
not the plaintext size).

The plaintext size of an entry is only filled in while borg is running, for the
chunks it actually processes: by ``borg create`` when it adds or re-uses a chunk,
or when the files cache entries of a previous archive are loaded (their chunks
lists have the plaintext sizes). So code using the chunks index must be prepared
to see size == 0 and must not assume it is the real chunk size.

The chunks index is a HashIndex_.

.. _cache-memory-usage:

Indexes / Caches memory usage
-----------------------------

Here is the estimated memory usage of Borg - it's complicated::

  chunk_size ~= 2 ^ HASH_MASK_BITS  (for buzhash chunker, BLOCK_SIZE for fixed chunker)
  chunk_count ~= total_file_size / chunk_size

  chunks_index_usage = chunk_count * 100

  files_cache_usage = total_file_count * 230 + chunk_count * 6

  mem_usage ~= chunks_index_usage + files_cache_usage
             = chunk_count * 106 + total_file_count * 230

All units are Bytes.

The 100 Bytes per chunks index entry are the 80 Bytes of the entry itself plus
the overhead of the hash table it lives in (see HashIndex_): the keys/values
arrays are over-allocated by up to 30%, and the bucket table adds another 4 Bytes
per bucket at a load factor of 0.25 .. 0.5. So, depending on where between two
resizes the index currently is, the real value is somewhere between 88 and 120
Bytes per entry - 100 is a good average.

The files cache numbers are for CPython on a 64bit platform: the ~230 Bytes per file
cover the dict slot, the 32 Bytes path hash (as a python ``bytes`` object) and the
fixed part of the msgpacked value; the ~6 Bytes per chunk are one msgpacked 32bit
index into the chunks index.

Both data structures grow by re-allocating and copying, so there are short-time
peaks in memory usage while a resize happens (worst case about 2x the values
computed above for the structure being resized). Usually this does not happen for
all data structures at the same time, though.

It is assuming every chunk is referenced exactly once (if you have a lot of
duplicate chunks, you will have fewer chunks than estimated above).

It is also assuming that typical chunk size is 2^HASH_MASK_BITS (if you have
a lot of files smaller than this statistical medium chunk size, you will have
more chunks than estimated above, because 1 file is at least 1 chunk).

The chunks index and files cache are both implemented as hash tables (the chunks
index as a HashIndex_, the files cache as a python dict). A hash table must have a
significant amount of unused entries to be fast - the so-called load factor gives
the used/unused elements ratio.

E.g. backing up a total count of 1 Mi (IEC binary prefix i.e. 2^20) files with a total size of 1TiB.

a) with ``create --chunker-params buzhash,10,23,16,4095`` (custom):

  chunk_count = 16 Mi, chunks_index_usage = 1.56GiB, files_cache_usage = 0.32GiB

  mem_usage  =  1.9GiB

b) with ``create --chunker-params buzhash,19,23,21,4095`` (default):

  chunk_count = 512 Ki, chunks_index_usage = 0.05GiB, files_cache_usage = 0.23GiB

  mem_usage  =  0.28GiB

.. note:: There is also the ``--files-cache=disabled`` option to disable the files cache.
   You'll save some memory, but it will need to read / chunk all the files as
   it can not skip unmodified files then.

.. _internals_hashindex:

HashIndex
---------

The chunks index is implemented on top of ``borghash.HashTableNT``, which comes from
the separate `borghash <https://github.com/borgbackup/borghash>`_ package (Cython).
``borg.hashindex.ChunkIndex`` only adds the borg specific parts on top of it: the
``ChunkIndexEntry`` namedtuple / struct format and the handling of the system flags.

``HashTableNT`` packs/unpacks the namedtuple value to/from ``bytes`` using a
``struct.Struct`` and delegates the actual storage to ``borghash.HashTable``, which
is a fixed key size / fixed value size ``bytes -> bytes`` mapping.

Internally, ``HashTable`` is not one, but three arrays:

- the *bucket table*, an array of ``uint32_t`` indexes into the keys/values arrays.
  ``0xffffffff`` marks an empty bucket, ``0xfffffffe`` marks a deleted bucket
  (tombstone); everything ``>= 0xffffff00`` is reserved, so the usable index range
  (and thus the maximum number of entries) is a bit below 4Gi.
- the *keys* array, holding ``key_size`` (32 for the chunks index) Bytes per entry.
- the *values* array, holding ``value_size`` (48 for the chunks index) Bytes per entry.

Keys and values are appended to their arrays in insertion order, so the index of a
key in the keys array is stable while the hash table is in memory. The files cache
uses that to "compress" chunk ids to 32bit numbers, see ``ChunkIndex.k_to_idx``.

The bucket table has only one slot per bucket, spreading hash collisions to the
following buckets. As a consequence the hash is just a start position for a linear
search. If a key is looked up that is not in the table, then the bucket table is
searched from the start position (the hash) until the first empty bucket is reached.

This particular mode of operation is open addressing with linear probing.

The bucket table is grown (by 2x) when the number of used buckets plus tombstones
exceeds 50% of its capacity, and shrunken (to 40%, but never below 1000 buckets)
when the number of used buckets drops below 10% of its capacity. So its load factor
usually is between 0.25 and 0.5. That is cheap, because a bucket is only 4 Bytes -
the bulk of the data is in the keys/values arrays, which are not hash tables and
thus do not need any unused space for speed. They are just grown by 1.3x whenever
they are full.

If an element is deleted, its bucket is marked with a tombstone (the keys/values
array slots are zeroed, but not reclaimed until the next rebuild). Tombstones are
only removed by resizing / rebuilding the bucket table. They present the same load
to the hash table as a real entry (recall that linear probing for an element not in
the index stops at the first empty bucket), which is why they count towards the load
factor that triggers the growth.

Data in a HashIndex is stored in little-endian format, which increases efficiency
for almost everyone, since basically no one uses big-endian processors any more.

HashIndex does not use a hashing function, because all keys (save manifest) are
outputs of a cryptographic hash or MAC and thus already have excellent distribution.
Thus, HashIndex simply uses the first 32 bits of the key as its "hash".

The on-disk format does not mirror the in-memory layout - neither the bucket table
nor the unused space of the keys/values arrays are written. A serialized HashIndex is:

- First, a header: the eight byte ASCII string "BORGHASH", an ``uint32`` format
  version and an ``uint32`` length of the metadata block (all little-endian).
- Second, the metadata block, a JSON object with the key size, value size, byte
  order, the value namedtuple's name / fields / struct format, the bucket table
  capacity and the number of entries ("used").
- Third, "used" times a (key, value) pair, without any padding or separators.

So the on-disk size is ``entries * (key_size + value_size)`` plus a small header,
i.e. exactly 80 Bytes per entry for the chunks index.

.. _data-encryption:

Encryption
----------

.. seealso:: The :ref:`borgcrypto` section for an in-depth review.

AEAD modes
~~~~~~~~~~

For new repositories, borg only uses modern AEAD ciphers: AES-OCB or CHACHA20-POLY1305.

For each borg invocation, a new sessionkey is derived from the borg key material
and the 48bit IV starts from 0 again. The cipher blocks of a chunk do not consume
IVs here (CHACHA20-POLY1305 counts them in its internal 32bit block counter, AES-OCB
derives the per-block offsets from the IV), so we just count up by 1 per chunk.

The encryption layout is best seen at the bottom of this diagram:

.. figure:: encryption-aead.png
    :figwidth: 100%
    :width: 100%

No special IV/counter management is needed here due to the use of session keys.

The 48 bit IV limits the number of **messages** (chunks and metadata objects) that we
encrypt with one session key to 2^48 - borg refuses to encrypt more rather than reusing
an IV. That is way more than needed: even if you only backed up 4kiB chunks (2^12B),
2^48 messages would be 2^(12+48)B == 1.2 exabytes of input data, meaning you would run
against other limitations (RAM, storage, time) way before that.

How much **data** we may encrypt with one session key is a different question, which is
not answered by the IV size, but by the security bounds of the ciphers, see below.

.. _aead_usage_limits:

AEAD usage limits
~~~~~~~~~~~~~~~~~

The relevant quantities are the number of encrypted messages (q), the amount of data
encrypted with one key and the number of forgery attempts (v, decryptions of tampered
data that borg refuses). ``p`` is the attacker's success probability we still consider
acceptable. See issue #6501 for the details and for the computations.

- **Number of messages** (both ciphers): limited to 2^48 per session key by the IV size,
  see above. This is never the binding limit for either cipher.
- **Data volume** (AES-OCB): the attacker's advantage grows with the square of the amount
  of data encrypted using one key: about ``6 * sigma^2 / 2^128``, ``sigma`` being the
  number of 128bit cipher blocks, including the authenticated header. RFC 7253 derives
  from this bound that one key should encrypt at most 2^48 blocks (4PiB), which
  corresponds to p == 2^-32. borg aims higher and starts a new session after 2^37 blocks
  (2TiB), which corresponds to p == 2^-51 per session key.

  CHACHA20-POLY1305 does not have such a limit at all: its confidentiality bound does not
  depend on the amount of data encrypted.
- **Forgery attempts** (CHACHA20-POLY1305): ``v <= p * 2^103 / (L' + 1)``, ``L'`` being
  the message length (payload plus authenticated header) in 128bit blocks. For borg's
  biggest messages, that is about 2^33 forgery attempts at p == 2^-50, so an attacker
  would have to make borg read more than 100PiB of tampered data. Note that this is
  counted over **all** session keys, so - unlike the data volume limit - it can not be
  improved by starting more sessions.

  For AES-OCB, the corresponding limit is much higher (its 128bit authentication tag
  gives a term in the order of ``v * L / 2^128``), so the CHACHA20-POLY1305 limit is the
  one to look at.

We do **not** count or enforce the forgery attempts limit, we just document it here:
a failed decryption means we got tampered or corrupted data and borg refuses it, usually
aborting the whole command (``borg check`` and archive listing keep going, but only to
report the damage). Getting anywhere near the limit computed above would require feeding
borg a lot more tampered data than any real repository will ever hold.

Starting a new session just means computing a new random session id and deriving a new
session key from it (and counting the IV from 0 again). That is cheap and it does not
need any special handling when reading, because the session id is part of every chunk
header. Because the advantages of the individual session keys just add up, frequent
session key changes also keep the total advantage low over the lifetime of a borg key.

.. _tagged_envelope:

Modes without encryption
~~~~~~~~~~~~~~~~~~~~~~~~

The ``authenticated-*`` and ``none-*`` modes do not encrypt: the payload of a repository
object slot (the compressed chunk data resp. the packed metadata, see `Repository objects`_)
is stored as-is. Every slot still carries a 32 byte tag::

    TYPE(1) + reserved(1) + tag(32) + payload

``TYPE`` is the key type byte (which identifies the mode, see ``KeyType``), ``reserved`` is
zero. The tag is computed over the envelope header, the AAD and the payload::

    aad_full = aad + chunk_id
    tag = MAC(tag_key, TYPE || reserved || len16_be(aad_full) || aad_full || payload)

``aad`` is what ``RepoObj`` authenticates alongside the payload: the object header prefix
(magic, format version, chunk id) and the slot tag (``M`` for meta, ``D`` for data), see
:ref:`pack-format`. Consequently, the tag detects modification of the payload, of the
metadata, of the object header, a swap of the meta and the data slot, and an object slice
taken from a different object. The length prefix keeps the boundary between the AAD and the
payload unambiguous.

There is no nonce, no session and no other state: the tag is deterministic. Two repositories
with the same key material therefore store byte-identical objects for identical input, which
allows deduplicating them on the filesystem level (e.g. with CoW/dedup tools).

The modes differ in the tag algorithm and in whether they have a key at all:

- ``authenticated-sha256`` / ``authenticated-blake3``: the tag is a **MAC** (HMAC-SHA256 resp.
  keyed BLAKE3), so only somebody who has the borg key can compute it - this detects malicious
  tampering, not just accidental corruption. The MAC key is derived from ``crypt_key``::

      tag_key = sha256(crypt_key + b"borg-repoobj-mac-hmac-sha256")[:32]   # authenticated-sha256
      tag_key = sha256(crypt_key + b"borg-repoobj-mac-blake3")[:32]        # authenticated-blake3

  It is deliberately not derived from ``id_key``: chunk ids are public, and related repositories
  share the id key (see ``borg repo-create --other-repo``), which must not enable them to forge
  each other's objects. ``--copy-crypt-key`` shares ``crypt_key`` and thus opts into producing
  byte-identical objects across the related repositories.
- ``none-sha256`` / ``none-blake3``: there is no key at all, so the tag is an **unkeyed** hash
  (plain SHA-256 resp. BLAKE3 over the same input), i.e. a checksum. It detects accidental
  corruption and reads that returned the wrong bytes, but anybody who modifies an object can
  recompute it - it is no protection against malicious tampering. For the same reason, the chunk
  ids of these modes are unkeyed hashes of the plaintext, which makes all repositories of such a
  mode dedup identically.

Legacy modes
~~~~~~~~~~~~

Old repositories (which used AES-CTR mode) are supported read-only to be able to
``borg transfer`` their archives to new repositories (which use AEAD modes).

AES-CTR mode is not supported for new repositories and the related code will be
removed in a future release.

The same applies to the borg 1.x ``none`` and ``authenticated`` modes: their envelope is just
the type byte followed by the payload, so nothing about an object is verified except the chunk
id over the plaintext. They were replaced by the tagged modes described above.

All modes
~~~~~~~~~

Encryption keys (and other secrets) are kept either in the keys directory on
the client ('keyfile' mode) or under the keys/ namespace in the repository
('repokey' mode) using the sha256 of the borg key content as the name.

In both cases, the secrets are generated from random and then encrypted by a
key derived from your passphrase (this happens on the client before the key
is stored as keyfile or repokey).

keyfile and repokey borg keys use the **same** format; only the storage location
differs. Borg finds the correct key by trying each key against the supplied
passphrase. See :ref:`borgcrypto_multiple_keys`.

The passphrase is passed through the ``BORG_PASSPHRASE`` environment variable
or prompted for interactive usage.

.. _key_files:

Key files
---------

.. seealso:: The :ref:`key_encryption` section for an in-depth review of the key encryption.

When initializing a repository with one of the "keyfile" encryption modes,
Borg creates an associated key file in the keys subdirectory of the borg config
directory (see :ref:`env_vars` for platform-specific default paths).

The same key is also used in the "repokey" modes, which store it in the repository.

The internal data structure is as follows:

version
  currently always an integer, 2

repository_id
  the ``id`` field in the ``config`` ``INI`` file of the repository.

crypt_key
  the initial key material used for the AEAD crypto (512 bits)

id_key
  the key used to MAC the plaintext chunk data to compute the chunk's id

chunk_seed
  the seed for the buzhash chunking table (signed 32 bit integer)

These fields are packed using msgpack_. The utf-8 encoded passphrase
is processed with argon2_ to derive a 256 bit key encryption key (KEK).

Then the KEK is used to encrypt and authenticate the packed data using
the chacha20-poly1305 AEAD cipher.

The result is stored in a another msgpack_ formatted as follows:

version
  currently always an integer, 1

salt
  random 256 bits salt used to process the passphrase

argon2_*
  some parameters for the argon2 kdf

algorithm
  the algorithms used to process the passphrase
  (currently the string ``argon2 chacha20-poly1305``)

data
  The encrypted, packed fields.

The resulting msgpack_ is then encoded using base64 and written to the
key file, wrapped using the standard ``textwrap`` module with a header.
The header is a single line with a MAGIC string, a space and a hexadecimal
representation of the repository id.

.. _data-compression:

Compression
-----------

Borg supports the following compression methods, each identified by a ctype value
in the range between 0 and 255 (and augmented by a one-byte clevel value for the
compression level):

- none (no compression, pass through data 1:1), identified by 0x00
- lz4 (low compression, but super fast), identified by 0x01
- zstd (level -128..22 offering a wide range: level 22 is higher compression and lower
  speed, level 1 is lower compression and high speed, and the negative "fast" levels
  trade still more compression for still more speed) - identified by 0x03
- zlib (level 0-9, level 0 is no compression [but still adding zlib overhead],
  level 1 is low, level 9 is high compression), identified by 0x05
- lzma (level 0-9, level 0 is low, level 9 is high compression), identified
  by 0x02.

The type byte is followed by a byte indicating the compression level. How that byte is
interpreted depends on the compression type: for zstd it is an ``int8_t``, so that the
negative levels fit (level -1 is stored as 255, -128 as 128). For all other types it is
an unsigned byte, with 255 meaning "no level applies" (as for none and lz4). Levels 1..22
occupy the same byte values either way, so zstd data written by older borg versions keeps
its meaning.

Speed:  none > lz4 > zlib > lzma, lz4 > zstd
Compression: lzma > zlib > lz4 > none, zstd > lz4

Be careful, higher compression levels might use a lot of resources (CPU/memory).

The overall speed of course also depends on the speed of your target storage.
If that is slow, using a higher compression level might yield better overall
performance. You need to experiment a bit. Maybe just watch your CPU load, if
that is relatively low, increase compression until 1 core is 70-100% loaded.

Even if your target storage is rather fast, you might see interesting effects:
while doing no compression at all (none) is a operation that takes no time, it
likely will need to store more data to the storage compared to using lz4.
The time needed to transfer and store the additional data might be much more
than if you had used lz4 (which is super fast, but still might compress your
data about 2:1). This is assuming your data is compressible (if you back up
already compressed data, trying to compress them at backup time is usually
pointless).

Compression is applied after deduplication, thus using different compression
methods in one repo does not influence deduplication.

See ``borg create --help`` about how to specify the compression level and its default.

Lock files (fslocking)
----------------------

Borg uses filesystem locks to get (exclusive or shared) access to the cache.

The locking system is based on renaming a temporary directory
to `lock.exclusive` (for
exclusive locks). Inside this directory, there is a file indicating
hostname, process id and thread id of the lock holder.

There is also a json file `lock.roster` that keeps a directory of all shared
and exclusive lockers.

If the process is able to rename a temporary directory (with the
host/process/thread identifier prepared inside it) in the resource directory
to `lock.exclusive`, it has the lock for it. If renaming fails
(because this directory already exists and its host/process/thread identifier
denotes a thread on the host which is still alive), lock acquisition fails.

The cache lock is usually in `~/.cache/borg/REPOID/lock.*`.

Locks (storelocking)
--------------------

To implement locking based on ``borgstore``, borg stores objects below locks/.

The objects contain:

- a timestamp when lock was created (or refreshed)
- host / process / thread information about lock owner
- lock type: exclusive or shared

Using that information, borg implements:

- lock auto-expiry: if a lock is old and has not been refreshed in time,
  it will be automatically ignored and deleted. the primary purpose of this
  is to get rid of stale locks by borg processes on other machines.
- lock auto-removal if the owner process is dead. the primary purpose of this
  is to quickly get rid of stale locks by borg processes on the same machine.

Breaking the locks
------------------

In case you run into troubles with the locks, you can use the ``borg break-lock``
command after you first have made sure that no Borg process is
running on any machine that accesses this resource. Be very careful, the cache
or repository might get damaged if multiple processes use it at the same time.

If there is an issue just with the repository lock, it will usually resolve
automatically (see above), just retry later.


Checksumming data structures
----------------------------

As detailed in the previous sections, Borg generates and stores various files
containing important meta data, such as the files cache.

Data corruption in the files cache could create incorrect archives, e.g. due
to wrong object IDs or sizes in the files cache.

Therefore, Borg calculates checksums when writing these files and tests checksums
when reading them. Checksums are generally 256-bit sha256 hashes.
Checksums are stored as hexadecimal ASCII strings.

For compatibility, checksums are not required and absent checksums do not trigger errors.
The mechanisms have been designed to avoid false-positives when various Borg
versions are used alternately on the same repositories.

Checksums are a data safety mechanism. They are not a security mechanism.

.. rubric:: Choice of algorithm

sha256 has been chosen for its wide availability on all platforms and hw acceleration on some.

Lower layer — file_integrity

{
    "algorithm": "SHA256",
    "digests": {
        "HashHeader": "eab6802590ba39e3...",
        "final": "e2a7f132fc2e8b24..."
    }
}

.. rubric:: Main cache files: chunks and files cache

The integrity data of the ``files`` cache is stored in the cache ``config``.

The ``[integrity]`` section is used:

.. code-block:: none

    [cache]
    version = 1
    repository = 3c4...e59
    manifest = 10e...21c
    timestamp = 2017-06-01T21:31:39.699514
    key_type = 2
    previous_location = /path/to/repo

    [integrity]
    manifest = 10e...21c
    files = {"algorithm": "SHA256", "digests": {"HashHeader": "eab...39e3", "final": "e2a...b24"}}

The manifest ID is duplicated in the integrity section due to the way all Borg
versions handle the config file. Instead of creating a "new" config file from
an internal representation containing only the data understood by Borg,
the config file is read in entirety (using the Python ConfigParser) and modified.
This preserves all sections and values not understood by the Borg version
modifying it.

Thus, if an older versions uses a cache with integrity data, it would preserve
the integrity section and its contents. If a integrity-aware Borg version
would read this cache, it would incorrectly report checksum errors, since
the older version did not update the checksums.

However, by duplicating the manifest ID in the integrity section, it is
easy to tell whether the checksums concern the current state of the cache.

Integrity errors are fatal in these files, terminating the program,
and are not automatically corrected at this time.


HardLinkManager and the hlid concept
------------------------------------

Dealing with hard links needs some extra care, implemented in borg within the HardLinkManager
class:

- At archive creation time, fs items with st_nlink > 1 indicate that they are a member of
  a group of hardlinks all pointing to the same inode. For such fs items, the archived item
  includes a hlid attribute (hardlink id), which is computed like H(st_dev, st_ino). Thus,
  if archived items have the same hlid value, they pointed to the same inode and form a
  group of hardlinks. Besides that, nothing special is done for any member of the group
  of hardlinks, meaning that e.g. for regular files, each archived item will have a
  chunks list.
- At extraction time, the presence of a hlid attribute indicates that there might be more
  hardlinks coming, pointing to the same content (inode), thus borg will remember the "hlid
  to extracted path" mapping, so it will know the correct path for extracting (hardlinking)
  the next hardlink of that group / with the same hlid.
- This symmetric approach (each item has all the information, e.g. the chunks list)
  simplifies dealing with such items a lot, especially for partial extraction, for the
  FUSE filesystem, etc.
- This is different from the asymmetric approach of old borg versions (< 2.0) and also from
  tar which have the concept of a main item (first hardlink, has the content) and content-less
  secondary items with by-name back references for each subsequent hardlink, causing lots
  of complications when dealing with them.

---

## File: docs/internals/frontends.rst

.. include:: ../global.rst.inc
.. highlight:: none

.. _json_output:

All about JSON: How to develop frontends
========================================

Borg does not have a public API on the Python level. That does not keep you from writing :code:`import borg`,
but does mean that there are no release-to-release guarantees on what you might find in that package, not
even for point releases (1.1.x), and there is no documentation beyond the code and the internals documents.

Borg does on the other hand provide an API on a command-line level. In other words, a frontend should
(for example) create a backup archive by invoking :ref:`borg_create`, provide command-line parameters/options
as needed, and parse JSON output from Borg.

Important: JSON output is expected to be UTF-8, but currently borg depends on the locale being configured
for that (must be a UTF-8 locale and *not* "C" or "ascii"), so that Python will choose to encode to UTF-8.
The same applies to any inputs read by borg, they are expected to be UTF-8 encoded also.

On POSIX systems, you can usually set environment vars to choose a UTF-8 locale:

::

    export LANG=en_US.UTF-8
    export LC_CTYPE=en_US.UTF-8


Another way to get Python's stdin/stdout/stderr streams to use UTF-8 encoding (without having
a UTF-8 locale / LANG / LC_CTYPE) is:

::

    export PYTHONIOENCODING=utf-8


See :issue:`2273` for more details.


Dealing with non-unicode byte sequences and JSON limitations
------------------------------------------------------------

Paths on POSIX systems can have arbitrary bytes in them (except 0x00 which is used as string terminator in C).

Nowadays, UTF-8 encoded paths (which decode to valid unicode) are the usual thing, but a lot of systems
still have paths from the past, when other, non-unicode codings were used. Especially old Samba shares often
have wild mixtures of misc. encodings, sometimes even very broken stuff.

borg deals with such non-unicode paths ("with funny/broken characters") by decoding such byte sequences using
UTF-8 coding and "surrogateescape" error handling mode, which maps invalid bytes to special unicode code points
(surrogate escapes). When encoding such a unicode string back to a byte sequence, the original byte sequence
will be reproduced exactly.

JSON should only contain valid unicode text without any surrogate escapes, so we can't just directly have a
surrogate-escaped path in JSON ("path" is only one example, this also affects other text-like content).

Borg deals with this situation like this (since borg 2.0):

For a valid unicode path (no surrogate escapes), the JSON will only have "path": path.

For a non-unicode path (with surrogate escapes), the JSON will have 2 entries:

- "path": path_approximation (pure valid unicode, all invalid bytes will show up as "?")
- "path_b64": path_bytes_base64_encoded (if you decode the base64, you get the original path byte string)

JSON users need to pick whatever suits their needs best. The suggested procedure (shown for "path") is:

- check if there is a "path_b64" key.
- if it is there, you will know that the original bytes path did not cleanly UTF-8-decode into unicode (has
  some invalid bytes) and that the string given by the "path" key is only an approximation, but not the precise
  path. if you need precision, you must base64-decode the value of "path_b64" and deal with the arbitrary byte
  string you'll get. if an approximation is fine, use the value of the "path" key.
- if it is not there, the value of the "path" key is all you need (the original bytes path is its UTF-8 encoding).


Logging
-------

Especially for graphical frontends it is important to be able to convey and reformat progress information
in meaningful ways. The ``--log-json`` option turns the stderr stream of Borg into a stream of JSON lines,
where each line is a JSON object. The *type* key of the object determines its other contents.

.. warning:: JSON logging requires successful argument parsing. Even with ``--log-json`` specified, a
    parsing error will be printed in plain text, because logging set-up happens after all arguments are
    parsed.

The following types are in use. Progress information is governed by the usual rules for progress information,
it is not produced unless ``--progress`` is specified.

archive_progress
    Output during operations creating archives (:ref:`borg_create` and :ref:`borg_recreate`).
    The following keys exist, each represents the current progress.

    original_size
        Original size of data processed so far (before compression and deduplication, may be empty/absent)
    compressed_size
        Compressed size (may be empty/absent)
    deduplicated_size
        Deduplicated size (may be empty/absent)
    nfiles
        Number of (regular) files processed so far (may be empty/absent)
    path
        Current path (may be empty/absent)
    time
        Unix timestamp (float)
    finished
        boolean indicating whether the operation has finished, only the last object for an *operation*
        can have this property set to *true*.

progress_message
    A message-based progress information with no concrete progress information, just a message
    saying what is currently being worked on.

    operation
        unique, opaque integer ID of the operation
    :ref:`msgid <msgid>`
        Message ID of the operation (may be *null*)
    finished
        boolean indicating whether the operation has finished, only the last object for an *operation*
        can have this property set to *true*.
    message
        current progress message (may be empty/absent)
    time
        Unix timestamp (float)

progress_percent
    Absolute progress information with defined end/total and current value.

    operation
        unique, opaque integer ID of the operation
    :ref:`msgid <msgid>`
        Message ID of the operation (may be *null*)
    finished
        boolean indicating whether the operation has finished, only the last object for an *operation*
        can have this property set to *true*.
    message
        A formatted progress message, this will include the percentage and perhaps other information
        (absent for finished == true)
    current
        Current value (always less-or-equal to *total*, absent for finished == true)
    info
        Array that describes the current item, may be *null*, contents depend on *msgid*
        (absent for finished == true)
    total
        Total value (absent for finished == true)
    time
        Unix timestamp (float)

file_status
    This is only output by :ref:`borg_create` and :ref:`borg_recreate` if ``--list`` is specified. The usual
    rules for the file listing applies, including the ``--filter`` option.

    status
        Single-character status as for regular list output
    path
        Path of the file system object

log_message
    Any regular log output invokes this type. Regular log options and filtering applies to these as well.

    time
        Unix timestamp (float)
    levelname
        Upper-case log level name (also called severity). Defined levels are: DEBUG, INFO, WARNING, ERROR, CRITICAL
    name
        Name of the emitting entity
    message
        Formatted log message
    :ref:`msgid <msgid>`
        Message ID, may be *null* or absent

See Prompts_ for the types used by prompts.

.. rubric:: Examples (reformatted, each object would be on exactly one line)
.. highlight:: json

:ref:`borg_extract` progress::

    {"message": "100.0% Extracting: src/borgbackup.egg-info/entry_points.txt",
     "current": 13000228, "total": 13004993, "info": ["src/borgbackup.egg-info/entry_points.txt"],
     "operation": 1, "msgid": "extract", "type": "progress_percent", "finished": false}
    {"message": "100.0% Extracting: src/borgbackup.egg-info/SOURCES.txt",
     "current": 13004993, "total": 13004993, "info": ["src/borgbackup.egg-info/SOURCES.txt"],
     "operation": 1, "msgid": "extract", "type": "progress_percent", "finished": false}
    {"operation": 1, "msgid": "extract", "type": "progress_percent", "finished": true}

:ref:`borg_create` file listing with progress::

    {"original_size": 0, "compressed_size": 0, "deduplicated_size": 0, "nfiles": 0, "type": "archive_progress", "path": "src"}
    {"type": "file_status", "status": "U", "path": "src/borgbackup.egg-info/entry_points.txt"}
    {"type": "file_status", "status": "U", "path": "src/borgbackup.egg-info/SOURCES.txt"}
    {"type": "file_status", "status": "d", "path": "src/borgbackup.egg-info"}
    {"type": "file_status", "status": "d", "path": "src"}
    {"original_size": 13176040, "compressed_size": 11386863, "deduplicated_size": 503, "nfiles": 277, "type": "archive_progress", "path": ""}

Internal transaction progress::

    {"message": "Saving files cache", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false}
    {"message": "Saving cache config", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false}
    {"message": "Saving index", "operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": false}
    {"operation": 2, "msgid": "cache.commit", "type": "progress_message", "finished": true}

A debug log message::

    {"message": "35 self tests completed in 0.08 seconds",
     "type": "log_message", "created": 1488278449.5575905, "levelname": "DEBUG", "name": "borg.archiver"}

Prompts
-------

Prompts assume a JSON form as well when the ``--log-json`` option is specified. Responses
are still read verbatim from *stdin*, while prompts are JSON messages printed to *stderr*,
just like log messages.

Prompts use the *question_prompt* and *question_prompt_retry* types for the prompt itself,
and *question_invalid_answer*, *question_accepted_default*, *question_accepted_true*,
*question_accepted_false* and *question_env_answer* types for information about
prompt processing.

The *message* property contains the same string displayed regularly in the same situation,
while the *msgid* property may contain a msgid_, typically the name of the
environment variable that can be used to override the prompt. It is the same for all JSON
messages pertaining to the same prompt.

.. rubric:: Examples (reformatted, each object would be on exactly one line)
.. highlight:: none

Providing an invalid answer::

    {"type": "question_prompt", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
     "message": "... Type 'YES' if you understand this and want to continue: "}
    incorrect answer  # input on stdin
    {"type": "question_invalid_answer", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING", "is_prompt": false,
     "message": "Invalid answer, aborting."}

Providing a false (negative) answer::

    {"type": "question_prompt", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
     "message": "... Type 'YES' if you understand this and want to continue: "}
    NO  # input on stdin
    {"type": "question_accepted_false", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
     "message": "Aborting.", "is_prompt": false}

Providing a true (affirmative) answer::

    {"type": "question_prompt", "msgid": "BORG_CHECK_I_KNOW_WHAT_I_AM_DOING",
     "message": "... Type 'YES' if you understand this and want to continue: "}
    YES  # input on stdin
    # no further output, just like the prompt without --log-json

Passphrase prompts
------------------

Passphrase prompts should be handled differently. Use the environment variables *BORG_PASSPHRASE*
and *BORG_NEW_PASSPHRASE* (see :ref:`env_vars` for reference) to pass passphrases to Borg, don't
use the interactive passphrase prompts.

When setting a new passphrase (:ref:`borg_repo-create`, :ref:`borg_key_change-passphrase`) normally
Borg prompts whether it should display the passphrase. This can be suppressed by setting
the environment variable *BORG_DISPLAY_PASSPHRASE* to *no*.

When "confronted" with an unknown repository, where the application does not know whether
the repository is encrypted, the following algorithm can be followed to detect encryption:

1. Set *BORG_PASSPHRASE* to gibberish (for example a freshly generated UUID4, which cannot
   possibly be the passphrase)
2. Invoke ``borg list repository ...``
3. If this fails, due the repository being encrypted and the passphrase obviously being
   wrong, you'll get an error with the *PassphraseWrong* msgid.

   The repository is encrypted, for further access the application will need the passphrase.

4. If this does not fail, then the repository is not encrypted.

Standard output
---------------

*stdout* is different and more command-dependent than logging. Commands like :ref:`borg_info`, :ref:`borg_create`
and :ref:`borg_list` implement a ``--json`` option which turns their regular output into a single JSON object.

Some commands, like :ref:`borg_list` and :ref:`borg_diff`, can produce *a lot* of JSON. Since many JSON implementations
don't support a streaming mode of operation, which is pretty much required to deal with this amount of JSON, these
commands implement a ``--json-lines`` option which generates output in the `JSON lines <https://jsonlines.org/>`_ format,
which is simply a number of JSON objects separated by new lines.

Dates are formatted according to ISO 8601 in local time. No explicit time zone is specified *at this time*
(subject to change). The equivalent strftime format string is '%Y-%m-%dT%H:%M:%S.%f',
e.g. ``2017-08-07T12:27:20.123456``.

The root object of '--json' output will contain at least a *repository* key with an object containing:

id
    The ID of the repository, normally 64 hex characters
location
    Canonicalized repository path, thus this may be different from what is specified on the command line
last_modified
    Date when the repository was last modified by the Borg client

The *encryption* key, if present, contains:

encryption
    Textual cipher / AE algorithm name (same as :ref:`borg_repo-create` ``--encryption`` names)
id_hash
    Textual id hash function name (same as :ref:`borg_repo-create` ``--id-hash`` names)
keyfile
    Path to the local key file used for access. Depending on the key location this may be absent.

The *cache* key, if present, contains:

path
    Path to the local repository cache
stats
    Object containing cache stats:

    total_chunks
        Number of chunks
    total_unique_chunks
        Number of unique chunks
    total_size
        Total uncompressed size of all chunks multiplied with their reference counts
    unique_size
        Uncompressed size of all chunks

.. highlight: json

Example *borg info* output::

    {
        "cache": {
            "path": "/home/user/.cache/borg/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
            "stats": {
                "total_chunks": 511533,
                "total_size": 22635749792,
                "total_unique_chunks": 54892,
                "unique_size": 2449675468
            }
        },
        "encryption": {
            "encryption": "aes256-ocb",
            "id_hash": "sha256"
        },
        "repository": {
            "id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
            "last_modified": "2017-08-07T12:27:20.789123",
            "location": "/home/user/testrepo"
        },
        "security_dir": "/home/user/.config/borg/security/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
        "archives": []
    }

Archive formats
+++++++++++++++

:ref:`borg_info` uses an extended format for archives, which is more expensive to retrieve, while
:ref:`borg_list` uses a simpler format that is faster to retrieve. Either return archives in an
array under the *archives* key, while :ref:`borg_create` returns a single archive object under the
*archive* key.

:ref:`borg_create` with ``--dry-run`` does not create an archive, so there is no *archive* key.
Instead, it returns *dry_run* (true) and a reduced *stats* object with *nfiles* and *original_size*,
both computed from file system metadata without reading the file contents.

Both formats contain a *name* key with the archive name, the *id* key with the hexadecimal archive ID,
and the *start* key with the start timestamp.

*borg info* and *borg create* further have:

end
    End timestamp
duration
    Duration in seconds between start and end in seconds (float)
stats
    Archive statistics (freshly calculated, this is what makes "info" more expensive)

    original_size
        Size of files and metadata before compression
    compressed_size
        Size after compression
    deduplicated_size
        Deduplicated size (against the current repository, not when the archive was created)
    nfiles
        Number of regular files in the archive
command_line
    Array of strings of the command line that created the archive

    The note about paths from above applies here as well.
chunker_params
    The chunker parameters the archive has been created with.

:ref:`borg_info` further has:

hostname
    Hostname of the creating host
username
    Name of the creating user
comment
    Archive comment, if any

Some keys/values are more expensive to compute than others (e.g. because it requires opening the archive,
not just the manifest). To optimize for speed, `borg list repo` does not determine these values except
when they are requested. The `--format` option is used for that (for normal mode as well as for `--json`
mode), so, to have the comment included in the json output, you will need:

::

    borg list repo --format "{name}{comment}" --json`


Example of a simple archive listing (``borg list --last 1 --json``)::

    {
        "archives": [
            {
                "id": "80cd07219ad725b3c5f665c1dcf119435c4dee1647a560ecac30f8d40221a46a",
                "name": "host-system-backup-2017-02-27",
                "start": "2017-08-07T12:27:20.789123"
            }
        ],
        "encryption": {
            "encryption": "aes256-ocb",
            "id_hash": "sha256"
        },
        "repository": {
            "id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
            "last_modified": "2017-08-07T12:27:20.789123",
            "location": "/home/user/repository"
        }
    }

The same archive with more information (``borg info --last 1 --json``)::

    {
        "archives": [
            {
                "chunker_params": [
                    "buzhash",
                    13,
                    23,
                    16,
                    4095
                ],
                "command_line": [
                    "/home/user/.local/bin/borg",
                    "create",
                    "/home/user/repository",
                    "..."
                ],
                "comment": "",
                "duration": 5.641542,
                "end": "2017-02-27T12:27:20.789123",
                "hostname": "host",
                "id": "80cd07219ad725b3c5f665c1dcf119435c4dee1647a560ecac30f8d40221a46a",
                "name": "host-system-backup-2017-02-27",
                "start": "2017-02-27T12:27:20.789123",
                "stats": {
                    "compressed_size": 1880961894,
                    "deduplicated_size": 2791,
                    "nfiles": 53669,
                    "original_size": 2400471280
                },
                "username": "user"
            }
        ],
        "cache": {
            "path": "/home/user/.cache/borg/0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
            "stats": {
                "total_chunks": 511533,
                "total_size": 22635749792,
                "total_unique_chunks": 54892,
                "unique_size": 2449675468
            }
        },
        "encryption": {
            "encryption": "aes256-ocb",
            "id_hash": "sha256"
        },
        "repository": {
            "id": "0cbe6166b46627fd26b97f8831e2ca97584280a46714ef84d2b668daf8271a23",
            "last_modified": "2017-08-07T12:27:20.789123",
            "location": "/home/user/repository"
        }
    }

File listings
+++++++++++++

Each archive item (file, directory, ...) is described by one object in the :ref:`borg_list` output.
Refer to the *borg list* documentation for the available keys and their meaning.

Example (excerpt) of ``borg list --json-lines``::

    {"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux", "target": "", "flags": null, "mtime": "2017-02-27T12:27:20.023407", "size": 0}
    {"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux/baz", "target": "", "flags": null, "mtime": "2017-02-27T12:27:20.585407", "size": 0}


Archive Differencing
++++++++++++++++++++

Each archive difference item (file contents, user/group/mode) output by :ref:`borg_diff` is represented by an *ItemDiff* object.
The properties of an *ItemDiff* object are:

path:
    The filename/path of the *Item* (file, directory, symlink).

changes:
    A list of *Change* objects describing the changes made to the item in the two archives. For example,
    there will be two changes if the contents of a file are changed, and its ownership are changed.

The *Change* object can contain a number of properties depending on the type of change that occurred.
If a 'property' is not required for the type of change, it is not output.
The possible properties of a *Change* object are:

type:
  The **type** property is always present. It identifies the type of change and will be one of these values:

  - *modified* - file contents changed.
  - *added* - the file was added.
  - *removed* - the file was removed.
  - *added directory* - the directory was added.
  - *removed directory* - the directory was removed.
  - *added link* - the symlink was added.
  - *removed link* - the symlink was removed.
  - *changed link* - the symlink target was changed.
  - *mode* - the file/directory/link mode was changed. Note - this could indicate a change from a
    file/directory/link type to a different type (file/directory/link), such as -- a file is deleted and replaced
    with a directory of the same name.
  - *owner* - user and/or group ownership changed.

size:
    If **type** == '*added*' or '*removed*', then **size** provides the size of the added or removed file.

added:
    If **type** == '*modified*' and chunk ids can be compared, then **added** and **removed** indicate the amount
    of data 'added' and 'removed'. If chunk ids can not be compared, then **added** and **removed** properties are
    not provided and the only information available is that the file contents were modified.

removed:
    See **added** property.

old_mode:
    If **type** == '*mode*', then **old_mode** and **new_mode** provide the mode and permissions changes.

new_mode:
    See **old_mode** property.

old_user:
    If **type** == '*owner*', then **old_user**, **new_user**, **old_group** and **new_group** provide the user
    and group ownership changes.

old_group:
    See **old_user** property.

new_user:
    See **old_user** property.

new_group:
    See **old_user** property.


Example (excerpt) of ``borg diff --json-lines``::

    {"path": "file1", "changes": [{"path": "file1", "changes": [{"type": "modified", "added": 17, "removed": 5}, {"type": "mode", "old_mode": "-rw-r--r--", "new_mode": "-rwxr-xr-x"}]}]}
    {"path": "file2", "changes": [{"type": "modified", "added": 135, "removed": 252}]}
    {"path": "file4", "changes": [{"type": "added", "size": 0}]}
    {"path": "file3", "changes": [{"type": "removed", "size": 0}]}


Archive Analysis
++++++++++++++++

:ref:`borg_analyze` ``--json`` emits the numbers of its text report as one object. All sizes are
byte values; the compression factor the text report shows is ``stored_size / source_size``.

Without ``--by-name``, the *dedup_size* and *hotspots* keys are present.

*dedup_size* describes the considered set of archives:

considered_archives
    Number of archives matching the archive filters
total_archives
    Number of non-deleted archives in the repository
whole_repository
    True if no archive was left over by the filters, so the considered set is the whole
    repository. Every referenced chunk is then trivially exclusive to the set, and the
    *exclusive* key is absent.
deduplicated
    Object with *source_size* and *stored_size*: the summed size of the union of chunks the
    considered archives reference, chunks shared within the set counted once
exclusive
    Object with *source_size* and *stored_size*: the chunks referenced only by the considered
    set, i.e. what deleting the whole set would free. Absent if *whole_repository* is true.
unreferenced
    Object with *stored_size* and *chunks*: the chunks no non-deleted archive references, which
    ``borg compact`` could free. Their source size is not known, as it is only recorded in the
    archives referencing a chunk.
total_chunks
    Number of chunks in the repository chunk index
missing_chunks
    Number of chunks referenced by an archive but absent from the repository chunk index

*hotspots* is a list of objects with *path* (directory path) and *size* (bytes of chunks added or
removed in that directory between consecutive archives), busiest directory first. It is ``null``
if fewer than two archives matched, as hot spots need at least two archives to compare.

With ``--by-name``, the *by_name* key is present instead, decomposing the whole repository:

archives
    Number of non-deleted archives in the repository
names
    List of objects with *name*, *archives* (number of archives with that name), *source_size*
    and *stored_size*. The sizes are what is exclusive to that name: no archive of another name
    references those chunks. Biggest *stored_size* first.
shared
    Object with *source_size* and *stored_size*: the chunks referenced by two or more names
unreferenced
    As above
total
    Object with *archives*, *source_size* and *stored_size*. Each chunk is counted in exactly one
    of *names*, *shared* and *unreferenced*, so the *names* and *shared* sizes add up to *total*.
total_chunks, missing_chunks
    As above

Example of ``borg analyze -a 'sh:userA-*' --json``::

    {
        "dedup_size": {
            "considered_archives": 2,
            "deduplicated": {"source_size": 3000, "stored_size": 3536},
            "exclusive": {"source_size": 2000, "stored_size": 3338},
            "missing_chunks": 0,
            "total_archives": 3,
            "total_chunks": 13,
            "unreferenced": {"chunks": 0, "stored_size": 0},
            "whole_repository": false
        },
        "encryption": {"encryption": "aes256-ocb", "id_hash": "sha256"},
        "hotspots": [{"path": "home/user/src", "size": 1000}],
        "repository": {
            "id": "06e4027d32f8eae8333f8fe06b1c2c46bf12f22ad10bd4d04a0f30751a26d77b",
            "last_modified": "2026-08-01T22:46:05.886533",
            "location": "/home/user/repository"
        }
    }


.. _msgid:

Message IDs
-----------

Message IDs are strings that essentially give a log message or operation a name, without actually using the
full text, since texts change more frequently. Message IDs are unambiguous and reduce the need to parse
log messages.

Assigned message IDs and related error RCs (exit codes) are:

.. See scripts/errorlist.py; this is slightly edited.

Errors
    Error rc: 2 traceback: no
        Error: {}
    ErrorWithTraceback rc: 2 traceback: yes
        Error: {}

    Buffer.MemoryLimitExceeded rc: 2 traceback: no
        Requested buffer size {} is above the limit of {}.
    EfficientCollectionQueue.SizeUnderflow rc: 2 traceback: no
        Could not pop_front first {} elements, collection only has {} elements..
    RTError rc: 2 traceback: no
        Runtime Error: {}

    CancelledByUser rc: 3 traceback: no
        Cancelled by user.

    CommandError rc: 4 traceback: no
        Command Error: {}
    PlaceholderError rc: 5 traceback: no
        Formatting Error: "{}".format({}): {}({})
    InvalidPlaceholder rc: 6 traceback: no
        Invalid placeholder "{}" in string: {}

    Repository.AlreadyExists rc: 10 traceback: no
        A repository already exists at {}.
    Repository.CheckNeeded rc: 12 traceback: yes
        Inconsistency detected. Please run "borg check {}".
    Repository.DoesNotExist rc: 13 traceback: no
        Repository {} does not exist.
    Repository.InsufficientFreeSpaceError rc: 14 traceback: no
        Insufficient free space to complete transaction (required: {}, available: {}).
    Repository.InvalidRepository rc: 15 traceback: no
        {} is not a valid repository. Check repo config.
    Repository.InvalidRepositoryConfig rc: 16 traceback: no
        {} does not have a valid configuration. Check repo config [{}].
    Repository.ObjectNotFound rc: 17 traceback: yes
        Object with key {} not found in repository {}.
    Repository.ParentPathDoesNotExist rc: 18 traceback: no
        The parent path of the repo directory [{}] does not exist.
    Repository.PathAlreadyExists rc: 19 traceback: no
        There is already something at {}.
    Repository.PathPermissionDenied rc: 21 traceback: no
        Permission denied to {}.

    MandatoryFeatureUnsupported rc: 25 traceback: no
        Unsupported repository feature(s) {}. A newer version of borg is required to access this repository.
    NoManifestError rc: 26 traceback: no
        Repository has no manifest.
    UnsupportedManifestError rc: 27 traceback: no
        Unsupported manifest envelope. A newer version is required to access this repository.

    Archive.AlreadyExists rc: 30 traceback: no
        Archive {} already exists
    Archive.DoesNotExist rc: 31 traceback: no
        Archive {} does not exist
    Archive.IncompatibleFilesystemEncodingError rc: 32 traceback: no
        Failed to encode filename "{}" into file system encoding "{}". Consider configuring the LANG environment variable.

    KeyfileInvalidError rc: 40 traceback: no
        Invalid key data for repository {} found in {}.
    KeyfileMismatchError rc: 41 traceback: no
        Mismatch between repository {} and key file {}.
    KeyfileNotFoundError rc: 42 traceback: no
        No key file for repository {} found in {}.
    NotABorgKeyFile rc: 43 traceback: no
        This file is not a borg key backup, aborting.
    RepoKeyNotFoundError rc: 44 traceback: no
        No key entry found in the config of repository {}.
    RepoIdMismatch rc: 45 traceback: no
        This key backup seems to be for a different backup repository, aborting.
    UnencryptedRepo rc: 46 traceback: no
        Key management not available for unencrypted repositories.
    UnknownKeyType rc: 47 traceback: no
        Key type {0} is unknown.
    UnsupportedPayloadError rc: 48 traceback: no
        Unsupported payload type {}. A newer version is required to access this repository.
    UnsupportedKeyFormatError rc: 49 traceback:no
        Your borg key is stored in an unsupported format. Try using a newer version of borg.


    NoPassphraseFailure rc: 50 traceback: no
        can not acquire a passphrase: {}
    PasscommandFailure rc: 51 traceback: no
        passcommand supplied in BORG_PASSCOMMAND failed: {}
    PassphraseWrong rc: 52 traceback: no
        passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND or via BORG_PASSPHRASE_FD is incorrect.
    PasswordRetriesExceeded rc: 53 traceback: no
        exceeded the maximum password retries

    Cache.CacheInitAbortedError rc: 60 traceback: no
        Cache initialization aborted
    Cache.EncryptionMethodMismatch rc: 61 traceback: no
        Repository encryption method changed since last access, refusing to continue
    Cache.RepositoryAccessAborted rc: 62 traceback: no
        Repository access aborted
    Cache.RepositoryIDNotUnique rc: 63 traceback: no
        Cache is newer than repository - do you have multiple, independently updated repos with same ID?
    Cache.RepositoryReplay rc: 64 traceback: no
        Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)

    LockError rc: 70 traceback: no
        Failed to acquire the lock {}.
    LockErrorT rc: 71 traceback: yes
        Failed to acquire the lock {}.
    LockFailed rc: 72 traceback: yes
        Failed to create/acquire the lock {} ({}).
    LockTimeout rc: 73 traceback: no
        Failed to create/acquire the lock {} (timeout).
    NotLocked rc: 74 traceback: yes
        Failed to release the lock {} (was not locked).
    NotMyLock rc: 75 traceback: yes
        Failed to release the lock {} (was/is locked, but not by me).

    ConnectionClosed rc: 80 traceback: no
        Connection closed by remote host
    ConnectionClosedWithHint rc: 81 traceback: no
        Connection closed by remote host. {}
    InvalidRPCMethod rc: 82 traceback: no
        RPC method {} is not valid
    PathNotAllowed rc: 83 traceback: no
        Repository path not allowed: {}
    RemoteRepository.RPCServerOutdated rc: 84 traceback: no
        Borg server is too old for {}. Required version {}
    UnexpectedRPCDataFormatFromClient rc: 85 traceback: no
        Borg {}: Got unexpected RPC data format from client.
    UnexpectedRPCDataFormatFromServer rc: 86 traceback: no
        Got unexpected RPC data format from server:
        {}
    ConnectionBrokenWithHint rc: 87 traceback: no
        Connection to remote host is broken. {}

    IntegrityError rc: 90 traceback: yes
        Data integrity error: {}
    FileIntegrityError rc: 91 traceback: yes
        File failed integrity check: {}
    DecompressionError rc: 92 traceback: yes
        Decompression error: {}


Warnings
    BorgWarning rc: 1
        Warning: {}
    BackupWarning rc: 1
        {}: {}

    FileChangedWarning rc: 100
        {}: file changed while we backed it up
    IncludePatternNeverMatchedWarning rc: 101
        Include pattern '{}' never matched.
    BackupError rc: 102
        {}: backup error
    BackupRaceConditionError rc: 103
        {}: file type or inode changed while we backed it up (race condition, skipped file)
    BackupOSError rc: 104
        {}: {}
    BackupPermissionError rc: 105
        {}: {}
    BackupIOError rc: 106
        {}: {}
    BackupFileNotFoundError rc: 107
        {}: {}
    BackupTimeoutError rc: 111
        {}: {}
    BackupBrokenSymlinkError rc: 112
        {}: {}

Operations
    - cache.begin_transaction
    - cache.download_chunks, appears with ``borg create --no-cache-sync``
    - cache.commit
    - cache.sync

      *info* is one string element, the name of the archive currently synced.
    - repository.compact_segments
    - repository.replay_segments
    - repository.check
    - check.verify_data
    - check.rebuild_manifest
    - check.rebuild_refcounts
    - extract

      *info* is one string element, the name of the path currently extracted.
    - extract.permissions
    - archive.delete
    - archive.calc_stats
    - prune
    - upgrade.convert_segments

Prompts
    BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK
        For "Warning: Attempting to access a previously unknown unencrypted repository"
    BORG_RELOCATED_REPO_ACCESS_IS_OK
        For "Warning: The repository at location ... was previously located at ..."
    BORG_CHECK_I_KNOW_WHAT_I_AM_DOING
        For "This is a potentially dangerous function..." (check --repair)
    BORG_DELETE_I_KNOW_WHAT_I_AM_DOING
        For "You requested to DELETE the repository completely *including* all archives it contains:"

---

## File: docs/internals/security.rst

.. include:: ../global.rst.inc

.. somewhat surprisingly the "bash" highlighter gives nice results with
   the pseudo-code notation used in the "Encryption" section.

.. highlight:: bash

========
Security
========

.. _borgcrypto:

Cryptography in Borg
====================

.. _attack_model:

Attack model
------------

The attack model of Borg is that the environment of the client process
(e.g. ``borg create``) is trusted and the repository (server) is not. The
attacker has any and all access to the repository, including interactive
manipulation (man-in-the-middle) for remote repositories.

Furthermore, the client environment is assumed to be persistent across
attacks (practically this means that the security database cannot be
deleted between attacks).

Under these circumstances Borg guarantees that the attacker cannot

1. modify the data of any archive without the client detecting the change
2. rename or add an archive without the client detecting the change
3. recover plain-text data
4. recover definite (heuristics based on access patterns are possible)
   structural information such as the object graph (which archives
   refer to what chunks)

The attacker can always impose a denial of service by definition (they could
block connections to the repository, or delete it partly or entirely).


.. _security_structural_auth:

Structural Authentication
-------------------------

Borg is fundamentally based on an object graph structure (see :ref:`internals`),
where the root objects are the archives.

Borg follows the `Horton principle`_, which states that
not only the message must be authenticated, but also its meaning (often
expressed through context), because every object used is referenced by a
parent object through its object ID up to the archive list entry. The object ID in
Borg is a MAC of the object's plaintext, therefore this ensures that
an attacker cannot change the context of an object without forging the MAC.

In other words, the object ID itself only authenticates the plaintext of the
object and not its context or meaning. The latter is established by a different
object referring to an object ID, thereby assigning a particular meaning to
an object. For example, an archive item contains a list of object IDs that
represent packed file metadata. On their own, it's not clear that these objects
would represent what they do, but by the archive item referring to them
in a particular part of its own data structure assigns this meaning.

This results in a directed acyclic graph of authentication from the archive
list entry to the data chunks of individual files.

Above used to be all for borg 1.x and was the reason why it needed the
tertiary authentication mechanism (TAM) for manifest and archives.

borg 2 now stores the ro_type ("meaning") of a repo object's data into that
object's metadata (like e.g.: manifest vs. archive vs. user file content data).
When loading data from the repo, borg verifies that the type of object it got
matches the type it wanted. borg 2 does not use TAMs any more.

As both the object's metadata and data are authenticated and also bound to
the object ID (via giving the ID as AAD), there is no way an attacker (without
access to the borg key) could change the type of the object or move content
to a different object ID. This holds for the AEAD encryption modes (where the
AEAD tag authenticates them) as well as for the ``authenticated-*`` modes
(where a MAC does, see :ref:`tagged_envelope`).

It does **not** hold for the ``none-*`` modes: they have no key, so their objects
carry an unkeyed checksum rather than a MAC, and an attacker who modifies an
object can simply recompute it. What still constrains an attacker there is the
object ID being the (unkeyed) hash of the plaintext: the content of an existing
object can not be replaced without the ID no longer matching. But the object's
metadata, the archives list and the manifest are not anchored to anything secret,
so a ``none-*`` repository provides no tamper protection - only detection of
accidental corruption.

This effectively 'anchors' each archive to the key, which is controlled by the
client, thereby anchoring the DAG starting from the archives list entry,
making it impossible for an attacker to add or modify any part of the
DAG without Borg being able to detect the tampering.

Please note that removing an archive by removing an entry from archives/*
is possible and is done by ``borg delete`` and ``borg prune`` within their
normal operation. An attacker could also remove some entries there, but, due to
encryption, would not know what exactly they are removing. An attacker with
repository access could also remove other parts of the repository or the whole
repository, so there is not much point in protecting against archive removal.

The borg 1.x way of having the archives list within the manifest chunk was
problematic as it required a read-modify-write operation on the manifest,
requiring a lock on the repository. We want to try less locking and more
parallelism in future.

Passphrase notes
----------------

Note that when using BORG_PASSPHRASE the attacker cannot swap the *entire*
repository against a new repository with e.g. repokey mode and no passphrase,
because Borg will abort access when BORG_PASSPHRASE is incorrect.

However, interactively a user might not notice this kind of attack
immediately, if she assumes that the reason for the absent passphrase
prompt is a set BORG_PASSPHRASE. See issue :issue:`2169` for details.

.. _security_encryption:

Encryption
----------

AEAD modes
~~~~~~~~~~

Modes: ``--encryption (aes256-ocb|chacha20-poly1305)`` plus
``--id-hash (sha256|blake3)``

Supported: borg 2.0+

The cipher is selected by ``--encryption`` (see :ref:`borg_repo-create`), the
key storage location (repokey or keyfile) by ``--key-location``, and the chunk
ID hash function by ``--id-hash`` — these three are orthogonal.

Encryption with these modes is based on AEAD ciphers (authenticated encryption
with associated data) and session keys.

Depending on the chosen mode different AEAD ciphers are used:

- AES-256-OCB - super fast, single-pass algorithm IF you have hw accelerated AES.
- chacha20-poly1305 - very fast, purely software based AEAD cipher.

The chunk ID is derived via a MAC over the plaintext (mac key taken from borg key):

- HMAC-SHA256 (``--id-hash sha256``) - super fast IF you have hw accelerated SHA256 (see section "Encryption" below).
- keyed BLAKE3 (``--id-hash blake3``) - very fast, purely software based algorithm.

For each borg invocation, a new session id is generated by `os.urandom`_.

From that session id, the initial key material (ikm, taken from the borg key)
and an application and cipher specific salt, borg derives a session key using a
"one-step KDF" based on just sha256.

For each session key, IVs (nonces) are generated by a counter which increments for
each encrypted message.

Session::

    sessionid = os.urandom(24)
    domain = "borg-session-key-CIPHERNAME"
    sessionkey = sha256(crypt_key + sessionid + domain)
    message_iv = 0

Encryption::

    id = MAC(id_key, data)
    compressed = compress(data)

    header = type-byte || 00h || message_iv || sessionid
    aad = id || header
    message_iv++
    encrypted, auth_tag = AEAD_encrypt(session_key, message_iv, compressed, aad)
    authenticated = header || auth_tag || encrypted

Decryption::

    # Given: input *authenticated* data and a *chunk-id* to assert
    type-byte, past_message_iv, past_sessionid, auth_tag, encrypted = SPLIT(authenticated)

    ASSERT(type-byte is correct)

    domain = "borg-session-key-CIPHERNAME"
    past_key = sha256(crypt_key + past_sessionid + domain)

    decrypted = AEAD_decrypt(past_key, past_message_iv, authenticated)

    decompressed = decompress(decrypted)

Notable:

- More modern and often faster AEAD ciphers instead of self-assembled stuff.
- Due to the usage of session keys, which just start at 0 per session, IVs (nonces)
  do not need long-term special care here as they did for the legacy encryption modes.
- The session key is also changed within a borg invocation if we encrypted a lot of data
  with it (AES-OCB only, see :ref:`aead_usage_limits`). Reading is not affected by this,
  because the session id is part of every message.
- The id is now also input into the authentication tag computation.
  This strongly associates the id with the written data (== associates the key with
  the value). When later reading the data for some id, authentication will only
  succeed if what we get was really written by us for that id.
- Because of that, additionally verifying ``id == MAC(id_key, decompressed)`` after
  decryption is optional for these modes: it is not what protects against a malicious
  repository (the authentication tag does that), it only detects chunks whose content
  does not match their id - which only a malicious or compromised borg client that had
  the borg key could have written. Borg therefore does not verify that on every read by
  default, see the ``BORG_ASSERT_ID`` environment variable for details and for the places
  where borg verifies it by default (e.g. ``borg check --verify-data``, which is the
  recommended periodic audit for this).


Authenticated modes

Modes: ``--encryption none-(sha256|blake3)``

Supported: borg 2.0+

These modes have no key at all: they neither encrypt nor authenticate. Every repository
object slot carries an *unkeyed* checksum (see :ref:`tagged_envelope`), which detects
accidental corruption - bad storage hardware, a truncated write, a read that returned the
wrong bytes - before the data is used. It is not a protection against an attacker: whoever
modifies an object can recompute the checksum, and the chunk IDs are unkeyed hashes as well.

You are advised not to use these modes. Use ``authenticated-*`` instead if you do not want
your data encrypted but do want to detect tampering; it is the same thing plus a key.

Legacy modes
~~~~~~~~~~~~

Modes: ``--encryption (repokey|keyfile)[-blake2]``, ``--encryption (none|authenticated)``

Supported: borg < 2.0

These were the AES-CTR based modes in previous borg versions, with the chunk ID
derived via HMAC-SHA256 or (in the ``-blake2`` variants) Blake2b. ``blake2b`` is
only used by these legacy modes; new repositories use ``sha256`` or ``blake3``
(see above).

The borg 1.x ``none`` and ``authenticated`` modes belong here, too: their repository
objects have no tag at all, so nothing about an object is verified except the chunk ID
over the plaintext - not even the object's metadata. They were replaced by the modes
described above, which cover metadata and object header as well.

borg 2.0 does not support creating new repos using these modes,
but ``borg transfer`` can still read such existing repos.


.. _key_encryption:

Offline key security
--------------------

Borg cannot secure the key material while it is running, because the keys
are needed in plain to decrypt/encrypt repository objects.

For offline storage of the encryption keys they are encrypted with a
user-chosen passphrase.

A 256 bit key encryption key (KEK) is derived from the passphrase
using argon2_ with a random 256 bit salt. The KEK is then used
to Encrypt-*then*-MAC a packed representation of the keys using the
chacha20-poly1305 AEAD cipher and a constant IV == 0.
The ciphertext is then converted to base64.

This base64-encoded *borg key* is then stored in the key file or under the
repository's ``keys/`` namespace (keyfile and repokey modes respectively), named
by the sha256 of its content.

The use of a constant IV is secure because an identical passphrase will
result in a different derived KEK for every key encryption due to the salt.

.. _borgcrypto_multiple_keys:

Multiple borg keys

The buzhash chunkers chunk according to the input data, the chunker's
parameters and secret key material (which all influence the chunk boundary
positions).

Secret key material:

- "buzhash": chunker seed (32bits), used for XORing the hardcoded buzhash table
- "buzhash64": bh64_key (256bits) is derived from ID key, used to cryptographically
  generate the table.

Small files below some specific threshold (default: 512 KiB) result in only one
chunk (identical content / size as the original file), bigger files result in
multiple chunks.

fixed chunker
~~~~~~~~~~~~~

This chunker yields fixed sized chunks, with optional support of a differently
sized header chunk. The last chunk is not required to have the full block size
and is determined by the input file size.

Within our attack model, an attacker possessing a specific set of files which
he assumes that the victim also possesses (and backups into the repository)
could try a brute force fingerprinting attack based on the chunk sizes in the
repository to prove his assumption.

To make this more difficult, borg has an ``obfuscate`` pseudo compressor, that
will take the output of the normal compression step and tries to obfuscate
the size of that output. Of course, it can only **add** to the size, not reduce
it. Thus, the optional usage of this mechanism comes at a cost: it will make
your repository larger (ranging from a few percent larger [cheap] to ridiculously
larger [expensive], depending on the algorithm/params you wisely choose).

The output of the compressed-size obfuscation step will then be encrypted and
authenticated, as usual. Of course, using that obfuscation would not make any
sense without encryption. Thus, the additional data added by the obfuscator
are just 0x00 bytes, which is good enough because after encryption it will
look like random anyway.

To summarize, this is making size-based fingerprinting difficult:

- user-selectable chunker algorithm (and parametrization)
- for the buzhash chunker: secret, random per-repo chunker seed
- user-selectable compression algorithm (and level)
- optional ``obfuscate`` pseudo compressor with different choices
  of algorithm and parameters

Secret key usage against fingerprinting
---------------------------------------

Borg uses the borg key also for chunking and chunk ID generation to protect against fingerprinting.
As usual for borg's attack model, the attacker is assumed to have access to a borg repository.

The borg key includes a secret random chunk_seed which (together with the chunking algorithm)
determines the cutting places and thereby the length of the chunks cut. Because the attacker trying
a chunk length fingerprinting attack would use a different chunker secret than the borg setup being
attacked, they would not be able to determine the set of chunk lengths for a known set of files.

The borg key also includes a secret random id_key. The chunk ID generation is not just using a simple
cryptographic hash like sha256 (because that would be insecure as an attacker could see the hashes of
small files that result only in 1 chunk in the repository). Instead, borg uses keyed hash (a MAC,
e.g. HMAC-SHA256) to compute the chunk ID from the content and the secret id_key. Thus, an attacker
can't compute the same chunk IDs for a known set of small files to determine whether these are stored
in the attacked repository.

Stored chunk proximity
----------------------

Borg does not try to obfuscate order / proximity of files it discovers by
recursing through the filesystem. For performance reasons, we sort directory
contents in file inode order (not in file name alphabetical order), so order
fingerprinting is not useful for an attacker.

But, when new files are close to each other (when looking at recursion /
scanning order), the resulting chunks will be also stored close to each other
in the resulting repository segment file(s).

This might leak additional information for the chunk size fingerprinting
attack (see above).

## 2. Official Technical Reference & Guides (borgbackup/borgbackup.github.io)

# www.borgbackup.org

[](https://github.com/borgbackup/borgbackup.github.io/actions)

This is the project website of BorgBackup.

## New site

We made a new, georgeous site based on threejs and GSAP - enjoy (and don't miss the engage button).

Notes:
- its source code is a single file ``index3.html``.
- index3.html gets automatically copied to index.html by the github build workflow.

## Old site

The old site is still accessible at index_old.html.

All pages are meant to load very quickly with few or no external resources. All pages are generated from reStructuredText (reST).

HTML files are **never** committed to this ("source") branch. GitHub Actions automatically updates
the published ("master") branch. **Therefore pull requests should be submitted only to the "source" branch.**

To locally preview changes and update the HTML after editing an .rst file, run `make`. To clean all rendered HTML and remake, run `make clean all`. 
This requires a relatively recent Docutils release, since it uses rst2html5. To install rst2html5, run: `python3 -m pip install rst2html5`.

## Release docs

Release documentation is published on RTD and released as part of the official tarballs,
as well as distributed in most distribution packages. These are only valid for their respective releases.

The *release* docs are managed on https://readthedocs.org/projects/borgbackup and available at https://borgbackup.readthedocs.io/en/latest/ (RTD).

The *borgweb* manual is on RTD as well: https://borgweb.readthedocs.io/

Previously, the documentation was hosted directly here before the move to RTD.
There are still redirects (borgbackup/, borgweb/) to the RTD site here
to keep old URLs working for a while; however, that does not work with subpages.

To manage the documentation and releases, see the development guide at:
https://borgbackup.readthedocs.io/en/latest/development.html#building-the-docs-with-sphinx

## Setup

GitHub Actions builds the "source" branch, generates HTML, commits it, and
pushes it to the "master" branch with a GitHub deploy key.

GitHub Pages is used as the host, and the TLS certificate comes from Let's Encrypt (via GitHub).

More details: https://github.com/borgbackup/borgbackup.github.io/issues/45