sccache Documentation

File: README.md Copy Chapter

sccache - Shared Compilation Cache

sccache is a ccache-like compiler caching tool. It is used as a compiler wrapper and avoids compilation when possible, storing cached results either on local disk or in one of several cloud storage backends. Multi-level caching with automatic backfill is supported for hierarchical cache architectures (see Multi-Level Cache).

sccache includes support for caching the compilation of Assembler, C/C++ code, Rust, as well as NVIDIA's CUDA using nvcc, and clang, AMD's ROCm HIP.

sccache also provides icecream-style distributed compilation (automatic packaging of local toolchains) for all supported compilers (including Rust). The distributed compilation system includes several security features that icecream lacks such as authentication, transport layer encryption, and sandboxed compiler execution on build servers. See the distributed quickstart guide for more information.

sccache is also available as a GitHub Actions to facilitate the deployment using GitHub Actions cache.

Table of Contents (ToC)

• Installation
• Usage
• Build Requirements
• Build
• Separating caches between invocations
• Overwriting the cache
• Debugging
• Interaction with GNU make jobserver
• Known Caveats
• Storage Options
  • Multi-Level (Hierarchical Caching)
  • Local
  • S3
  • R2
  • Redis
  • Memcached
  • Google Cloud Storage
  • Azure
  • GitHub Actions
  • WebDAV (Ccache/Bazel/Gradle compatible)
  • Alibaba OSS
  • Tencent Cloud Object Storage

Installation Copy Chapter

There are prebuilt x86-64 binaries available for Windows, Linux (a portable binary compiled against musl), and macOS on the releases page. Several package managers also include sccache packages, you can install the latest release from source using cargo, or build directly from a source checkout.

macOS Copy Section

On macOS sccache can be installed via Homebrew:

brew install sccache

or via MacPorts:

sudo port install sccache

Windows Copy Section

On Windows, sccache can be installed via scoop:

scoop install sccache

or winget:

winget install Mozilla.sccache

Via cargo Copy Section

If you have a Rust toolchain installed you can install sccache using cargo. Note that this will compile sccache from source which is fairly resource-intensive. For CI purposes you should use prebuilt binary packages.

cargo install sccache --locked

Or, using cargo-binstall to install a prebuilt binary:

cargo binstall sccache

With Nix Copy Section

Sccache is available in nixpkgs, so if you don't need the latest version you can use that:

buildInputs = [ pkgs.sccache ];

We also provide a flake with an overlay for getting the latest version:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    sccache = {
      url = "github:mozilla/sccache";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, sccache, ... }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs {
        inherit system;
        overlays = [ sccache.overlays.default ];
      };
    in {
      devShells.${system}.default = pkgs.mkShell {
        buildInputs = [ pkgs.sccache ];
      };
    };
}

Or use it directly from the flake without the overlay:

nix run github:mozilla/sccache -- --help
nix shell github:mozilla/sccache

Usage Copy Chapter

Running sccache is like running ccache: prefix your compilation commands with it, like so:

sccache gcc -o foo.o -c foo.c

If you want to use sccache for caching Rust builds you can define build.rustc-wrapper in the
cargo configuration file. For example, you can set it globally
in $HOME/.cargo/config.toml by adding:

[build]
rustc-wrapper = "/path/to/sccache"

Note that you need to use cargo 1.40 or newer for this to work.

Alternatively you can use the environment variable RUSTC_WRAPPER:

export RUSTC_WRAPPER=/path/to/sccache
cargo build

sccache supports gcc, clang, MSVC, rustc, NVCC, NVC++, hipcc, and Wind River's diab compiler. Both gcc and msvc support Response Files, read more about their implementation here.

If you don't specify otherwise, sccache will use a local disk cache.

sccache works using a client-server model, where the server runs locally on the same machine as the client. The client-server model allows the server to be more efficient by keeping some state in memory. The sccache command will spawn a server process if one is not already running, or you can run sccache --start-server to start the background server process without performing any compilation.

By default sccache server will listen on 127.0.0.1:4226, you can specify environment variable SCCACHE_SERVER_PORT to use a different port or SCCACHE_SERVER_UDS to listen on unix domain socket. Abstract unix socket is also supported as long as the path is escaped following the format. For example:

% env SCCACHE_SERVER_UDS=$HOME/sccache.sock sccache --start-server # unix socket
% env SCCACHE_SERVER_UDS=\\x00sccache.sock sccache --start-server # abstract unix socket

You can run sccache --stop-server to terminate the server. It will also terminate after (by default) 10 minutes of inactivity.

Running sccache --show-stats will print a summary of cache statistics.

Some notes about using sccache with Jenkins are here.

To use sccache with cmake, provide the following command line arguments to cmake 3.4 or newer:

-DCMAKE_C_COMPILER_LAUNCHER=sccache
-DCMAKE_CXX_COMPILER_LAUNCHER=sccache

The process for using sccache with MSVC and cmake, depends on which version of cmake you're using. For versions of cmake 3.24 and earlier, to generate PDB files for debugging with MSVC, you can use the /Z7 option. Alternatively, the /Zi option together with /Fd can work if /Fd names a different PDB file name for each object file created. Note that CMake sets /Zi by default, so if you use CMake, you can use /Z7 by adding code like this in your CMakeLists.txt:

if(CMAKE_BUILD_TYPE STREQUAL "Debug")
  string(REPLACE "/Zi" "/Z7" CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG}")
  string(REPLACE "/Zi" "/Z7" CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG}")
elseif(CMAKE_BUILD_TYPE STREQUAL "Release")
  string(REPLACE "/Zi" "/Z7" CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE}")
  string(REPLACE "/Zi" "/Z7" CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE}")
elseif(CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo")
  string(REPLACE "/Zi" "/Z7" CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO}")
  string(REPLACE "/Zi" "/Z7" CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO}")
endif()

By default, sccache will fail your build if it fails to successfully communicate with its associated server. To have sccache instead gracefully failover to the local compiler without stopping, set the environment variable SCCACHE_IGNORE_SERVER_IO_ERROR=1.

For versions of cmake 3.25 and later, to compile with MSVC, you have to use the new CMAKE_MSVC_DEBUG_INFORMATION_FORMAT option, meant to configure the -Z7 flag. Additionally, you must set the cmake policy number 0141 to the NEW setting:

set(CMAKE_MSVC_DEBUG_INFORMATION_FORMAT "$<$<CONFIG:Debug,RelWithDebInfo>:Embedded>")
cmake_policy(SET CMP0141 NEW)

Example configuration where we automatically look for sccache in the PATH:

find_program(SCCACHE sccache REQUIRED)

set(CMAKE_C_COMPILER_LAUNCHER ${SCCACHE})
set(CMAKE_CXX_COMPILER_LAUNCHER ${SCCACHE})
set(CMAKE_MSVC_DEBUG_INFORMATION_FORMAT "$<$<CONFIG:Debug,RelWithDebInfo>:Embedded>")
cmake_policy(SET CMP0141 NEW)

Alternatively, if configuring cmake with MSVC on the command line, assuming that sccache is on the default search path:

cmake -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache -DCMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded -DCMAKE_POLICY_CMP0141=NEW [...]

And you can build code as usual without any additional flags in the command line, which is useful for IDEs.

Build Requirements Copy Chapter

sccache is a Rust program. Building it requires cargo (and thusrustc). sccache currently requires Rust 1.91.0. We recommend you install Rust via Rustup.

Build Copy Chapter

If you are building sccache for non-development purposes make sure you use cargo build --release to get optimized binaries:

cargo build --release [--no-default-features --features=s3|redis|gcs|memcached|azure|gha|webdav|oss|cos]

The list of features can be found in the Cargo.toml file, [features] section.

By default, sccache builds with support for all storage backends, but individual backends may be disabled by resetting the list of features and enabling all the other backends. Refer the Cargo Documentation for details on how to select features with Cargo.

Building portable binaries Copy Section

When building with the dist-server feature, sccache will depend on OpenSSL, which can be an annoyance if you want to distribute portable binaries. It is possible to statically link against OpenSSL using the openssl/vendored feature.

Linux

Build with cargo and use ldd to check that the resulting binary does not depend on OpenSSL anymore.

macOS

Build with cargo and use otool -L to check that the resulting binary does not depend on OpenSSL anymore.

Windows

On Windows, the binary might also depend on a few MSVC CRT DLLs that are not available on older Windows versions.

It is possible to statically link against the CRT using a .cargo/config.toml file with the following contents.

[target.x86_64-pc-windows-msvc]
rustflags = ["-Ctarget-feature=+crt-static"]

Build with cargo and use dumpbin /dependents to check that the resulting binary does not depend on MSVC CRT DLLs anymore.

When statically linking with OpenSSL, you will need Perl available in your $PATH.

Separating caches between invocations Copy Chapter

In situations where several different compilation invocations
should not reuse the cached results from each other,
one can set SCCACHE_C_CUSTOM_CACHE_BUSTER to a unique value
that'll be mixed into the hash.
MACOSX_DEPLOYMENT_TARGET and IPHONEOS_DEPLOYMENT_TARGET variables
already exhibit such reuse-suppression behaviour.
There are currently no such variables for compiling Rust.

Overwriting the cache Copy Chapter

In situations where the cache contains broken build artifacts, it can be necessary to overwrite the contents in the cache. That can be achieved by setting the SCCACHE_RECACHE environment variable.

Debugging Copy Chapter

You can set the SCCACHE_ERROR_LOG environment variable to a path and set SCCACHE_LOG to get the server process to redirect its logging there (including the output of unhandled panics, since the server sets RUST_BACKTRACE=1 internally).

SCCACHE_ERROR_LOG=/tmp/sccache_log.txt SCCACHE_LOG=debug sccache

You can also set these environment variables for your build system, for example

SCCACHE_ERROR_LOG=/tmp/sccache_log.txt SCCACHE_LOG=debug cmake --build /path/to/cmake/build/directory

Alternatively, if you are compiling locally, you can run the server manually in foreground mode by running SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 sccache, and send logging to stderr by setting the SCCACHE_LOG environment variable for example. This method is not suitable for CI services because you need to compile in another shell at the same time.

SCCACHE_LOG=debug SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 sccache

Interaction with GNU `make` jobserver Copy Chapter

sccache provides support for a GNU make jobserver. When the server is started from a process that provides a jobserver, sccache will use that jobserver and provide it to any processes it spawns. (If you are running sccache from a GNU make recipe, you will need to prefix the command with + to get this behavior.) If the sccache server is started without a jobserver present it will create its own with the number of slots equal to the number of available CPU cores.

This is most useful when using sccache for Rust compilation, as rustc supports using a jobserver for parallel codegen, so this ensures that rustc will not overwhelm the system with codegen tasks. Cargo implements its own jobserver (see the information on NUM_JOBS in the cargo documentation) for rustc to use, so using sccache for Rust compilation in cargo via RUSTC_WRAPPER should do the right thing automatically.

Normalizing Paths with `SCCACHE_BASEDIRS` Copy Chapter

By default, sccache requires absolute paths to match for cache hits. To enable cache sharing across different build directories, you can set SCCACHE_BASEDIRS to strip a base directory from paths before hashing:

export SCCACHE_BASEDIRS=/home/user/project

You can also specify multiple base directories by separating them by ; on Windows hosts and by : on any other operating system. When multiple directories are provided, the longest matching prefix is used:

export SCCACHE_BASEDIRS="/home/user/project:/home/user/workspace"

Path matching is case-insensitive on Windows and case-sensitive on other operating systems.

This is similar to ccache's CCACHE_BASEDIR and helps when:

• Building the same project from different directories
• Sharing cache between CI jobs with different checkout paths
• Multiple developers working with different username paths
• Working with multiple project checkouts simultaneously

Note: Only absolute paths are supported. Relative paths will prevent server from starting.

You can also configure this in the sccache config file:

# Single directory
basedirs = ["/home/user/project"]

# Or multiple directories
basedirs = ["/home/user/project", "/home/user/workspace"]

Known Caveats Copy Chapter

General Copy Section

• By default, absolute paths to files must match to get a cache hit. To work around this, use SCCACHE_BASEDIRS (see above) to normalize paths before hashing.

Rust Copy Section

• Crates that invoke the system linker cannot be cached. This includes bin, dylib, cdylib, and proc-macro crates. You may be able to improve compilation time of large bin crates by converting them to a lib crate with a thin bin wrapper.
• Incrementally compiled crates cannot be cached. By default, in the debug profile Cargo will use incremental compilation for workspace members and path dependencies. You can disable incremental compilation.

More details on Rust caveats

C++20 Modules Copy Section

sccache has partial support for C++20 named modules when using Clang. The following flags are supported:

• -fmodule-file= and -fmodule-file=<name>= - importing precompiled module interfaces
• -fmodule-output= - generating module interface output alongside object files
• --precompile - compiling module interface units
• -fmodules-reduced-bmi - generating reduced BMI files

The following module-related flags are not supported and will bypass the cache:

• -fmodules and -fcxx-modules - Clang header modules (not C++20 named modules)
• -fprebuilt-implicit-modules and -fprebuilt-module-path - implicit module discovery

GCC and MSVC C++20 modules are not yet supported. Compilations using -fmodules-ts (GCC) or /interface, /ifcOutput, etc. (MSVC) will bypass the cache.

User Agent Copy Section

• Requests sent to your storage option of choice will have a user agent header indicating the current sccache version, e.g. sccache/0.8.2.

Storage Options Copy Chapter

• Local
• S3
• R2
• Redis
• Memcached
• Google Cloud Storage
• Azure
• GitHub Actions
• WebDAV (Ccache/Bazel/Gradle compatible)
• Alibaba OSS
• Tencent Cloud Object Storage

File: docs/Architecture.md Copy Chapter

Sccache high level architecture

This schema shows at a high level how sccache decides whether to compile or to
reuse a cached result.

flowchart LR
      id1[[Environment variables]] --> hash
      id2[[Compiler binary]] --> hash
      id3[[Compiler arguments]] --> hash
      id5[[Files]] --> |  | hash
      Compile --> Upload
      Storage[(Storage)] --> | yes | Download
      hash([hash]) --> | exists? | Storage
      Storage --> | no | Compile
      Upload --> Storage

      classDef input fill:#4fc3f7,stroke:#01579b,color:#000
      classDef store fill:#ffb74d,stroke:#e65100,color:#000
      classDef action fill:#81c784,stroke:#1b5e20,color:#000
      class id1,id2,id3,id5 input
      class Storage store
      class hash,Compile,Upload,Download action

For more details about how hash generation works, see the caching documentation.

Direct mode (preprocessor cache) Copy Chapter

For C/C++, sccache can also cache the preprocessor result, so that a cache
lookup can skip preprocessing entirely. This is inspired by
ccache's direct mode
and is described in the local storage doc.

Before computing the object-cache key, sccache looks up a preprocessor cache
entry keyed on the source file (path + contents) and the preprocessor
arguments. That entry records every file included by the source. If the entry
exists and all of those included files are unchanged, sccache reuses it and
never runs the preprocessor; otherwise it preprocesses normally and stores a
fresh entry. This step is shown as the blue region in the diagrams below and is
enabled by default (it is skipped for non-C/C++, when disabled, or when certain
flags such as -Wp,* / -Xpreprocessor are present).

Execution modes Copy Chapter

The caching logic above is the same regardless of where it runs. What differs
is which process actually runs the compiler and talks to the storage backend.

sccache is split in two:

• a short-lived CLI process, spawned once per compiler invocation (e.g. by
  make -jN), and
• a long-lived daemon (the background server), which holds the storage
  backend and the accumulated statistics.

The two communicate over a local IPC connection. There are two ways to divide
the work between them.

Server-side mode (default) Copy Section

The CLI forwards the whole compilation to the daemon. The daemon runs the
direct-mode preprocessor cache lookup, computes the hash, looks up the object
cache, runs the compiler on a miss, and stores the result. It writes the output
files (e.g. the object file) to disk itself; only the output streams (stdout /
stderr) and the exit code travel back to the CLI.

sequenceDiagram
      participant B as Build (make -jN)
      participant C as sccache CLI
      participant D as sccache daemon
      participant S as Storage backend
      B->>C: sccache cc -c foo.c
      C->>D: Compile request (IPC)
      rect rgba(33, 150, 243, 0.18)
          note over D,S: Direct mode — preprocessor cache (C/C++)
          D->>S: lookup preprocessor entry (source + args)
          alt entry found & included files unchanged
              S-->>D: manifest → reuse, skip preprocessing
          else miss / stale / disabled
              D->>D: run preprocessor
              D->>S: store preprocessor entry
          end
      end
      rect rgba(76, 175, 80, 0.18)
          note over D,S: Object cache
          D->>S: lookup object (hash)
          alt cache hit
              S-->>D: cached object
          else cache miss
              D->>D: run compiler
              D->>S: store object
          end
      end
      D-->>C: CompileFinished (stdout / stderr / exit code)
      C-->>B: exit code + outputs

Client-side mode (`SCCACHE_CLIENT_SIDE`) Copy Section

The compile pipeline runs in the CLI process itself; the daemon is used only
as a shared gateway to the storage backend (and as a place to aggregate stats).

On startup the CLI performs a one-shot StorageHandshake to fetch the cache
metadata (cache mode, max size, basedirs, preprocessor-cache config). It then
runs the same pipeline locally — including direct mode — and forwards each
individual cache operation to the daemon over IPC. On the client side this is
implemented by IpcStorage, which implements the same Storage trait as every
other backend, so the rest of the compile pipeline is unchanged.

sequenceDiagram
      participant B as Build (make -jN)
      participant C as sccache CLI (IpcStorage)
      participant D as sccache daemon
      participant S as Storage backend
      B->>C: sccache cc -c foo.c
      C->>D: StorageHandshake (IPC)
      D-->>C: cache metadata (mode, max_size, basedirs, …)
      rect rgba(33, 150, 243, 0.18)
          note over C,S: Direct mode — preprocessor cache (C/C++)
          C->>D: StorageGetPreprocessorEntry (key)
          D->>S: lookup
          alt entry found & included files unchanged
              D-->>C: manifest → reuse, skip preprocessing
          else miss / stale / disabled
              D-->>C: miss
              C->>C: run preprocessor
              C->>D: StoragePutPreprocessorEntry (key, bytes)
              D->>S: store
          end
      end
      rect rgba(76, 175, 80, 0.18)
          note over C,S: Object cache
          C->>C: compute hash
          C->>D: StorageGetPath / StorageGetRaw (key)
          D->>S: lookup
          alt cache hit
              S-->>D: path / bytes
              D-->>C: path / bytes
          else cache miss
              D-->>C: miss
              C->>C: run compiler
              C->>D: StoragePutRaw (key, bytes)
              D->>S: store
          end
      end
      C->>D: RecordStats (delta)
      C-->>B: exit code + outputs

StorageGetPath lets the CLI read a cached entry straight off disk when the
backend exposes a local path; for backends that don't (S3, Redis, …) the CLI
falls back to fetching the raw bytes with StorageGetRaw. Direct-mode entries
are exchanged with StorageGetPreprocessorEntry / StoragePutPreprocessorEntry.
Because each CLI process accumulates its own statistics, it flushes them to the
daemon with RecordStats before exiting.

Client-side mode is enabled with the SCCACHE_CLIENT_SIDE environment variable
(or the client_side_mode config key). It is expected to become the only
supported configuration in the future, with server-side mode eventually
removed. It is currently mutually exclusive with:

• error logging (SCCACHE_ERROR_LOG): the client always logs to stderr, and
  multiple concurrent CLI processes would race on the log file, and
• distributed compilation (a configured scheduler URL).

If either is in use, the setting is ignored and sccache stays in server-side
mode.

File: docs/Azure.md Copy Chapter

Azure

sccache can store its cache in an Azure Blob Storage container using either a
shared-key connection string or Microsoft Entra ID (passwordless) authentication.
In both cases set SCCACHE_AZURE_BLOB_CONTAINER to the name of an existing
container — sccache will not create it for you.

Shared key (connection string) Copy Chapter

Set the SCCACHE_AZURE_CONNECTION_STRING environment variable to your storage
account connection string.

Microsoft Entra ID (passwordless) Copy Chapter

Use this when the storage account has shared-key access disabled. Instead of a
connection string, tell sccache which account to talk to with one of the
following (if both are set, SCCACHE_AZURE_ENDPOINT takes precedence):

• SCCACHE_AZURE_STORAGE_ACCOUNT — the storage account name. sccache builds the
  endpoint https://{account}.blob.core.windows.net.
• SCCACHE_AZURE_ENDPOINT — a full blob endpoint, for sovereign clouds
  (https://{account}.blob.core.usgovcloudapi.net,
  https://{account}.blob.core.chinacloudapi.cn) or custom DNS. Takes
  precedence over SCCACHE_AZURE_STORAGE_ACCOUNT. It must use https; plain
  http is accepted only for a loopback host.

SCCACHE_AZURE_CONNECTION_STRING and SCCACHE_AZURE_STORAGE_ACCOUNT /
SCCACHE_AZURE_ENDPOINT are mutually exclusive.

Azurite: the local emulator authenticates with its well-known shared key,
so point sccache at it with SCCACHE_AZURE_CONNECTION_STRING (the shared-key
section above), not the passwordless path — the latter would require Azurite's
non-default OAuth mode plus an ambient Entra credential.

Credentials are then resolved from the ambient environment, in this order:

1. Service principal — AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET.
2. Workload identity (AKS / Kubernetes / federated OIDC) — AZURE_TENANT_ID,
   AZURE_CLIENT_ID, AZURE_FEDERATED_TOKEN_FILE.
3. Managed identity via the instance-metadata endpoint (169.254.169.254) —
   Azure VM / VM Scale Sets and self-hosted CI agents running on them. No
   variables required; set AZURE_CLIENT_ID to select a specific user-assigned
   identity.

Managed-identity limitations. App Service, Functions, Container Apps, and
Container Instances expose managed identity only over the IDENTITY_ENDPOINT
protocol, which this backend does not use — prefer workload identity or a
service principal on those hosts. In any environment with an HTTP proxy
configured, add 169.254.169.254 to NO_PROXY, or the metadata request is
routed to the proxy and fails. Credentials resolve lazily, so a misconfigured
passwordless setup builds successfully and fails only on the first cache
access, not at server startup.

For sovereign clouds, AZURE_AUTHORITY_HOST is required for the service
principal and workload identity flows (it defaults to
https://login.microsoftonline.com): use https://login.microsoftonline.us
(US Gov) or https://login.partner.microsoftonline.cn (China). Managed identity
via IMDS does not need it. The identity needs a data-plane role on the container:
Storage Blob Data Contributor for READ_WRITE (the default), or Storage
Blob Data Reader when SCCACHE_AZURE_RW_MODE=READ_ONLY. sccache does not read
or log any AZURE_* value — they are consumed directly by the underlying storage
SDK when signing requests with an OAuth bearer token.

Note: the underlying storage SDK also honours the ambient
AZBLOB_ACCOUNT_KEY / AZBLOB_SAS_TOKEN variables, which take precedence over
Entra ID credentials. Unset them if you intend to authenticate passwordlessly.

Common options Copy Chapter

You can define a prefix that will be prepended to the keys of all cache objects
created and read within the container, effectively creating a scope. To do that
use the SCCACHE_AZURE_KEY_PREFIX environment variable. This can be useful when
sharing a container with another application.

The SCCACHE_AZURE_RW_MODE environment variable can be set to READ_ONLY to
make sccache use this backend in read-only mode. The default is READ_WRITE.

Important: The environment variables are only taken into account when the
server starts, i.e. only on the first run.

File: docs/Caching.md Copy Chapter

How caching works

To know if the storage contains the artifact we need, we are
computing some hashes to make sure the input is the same.

Because the configuration and environment matter, the hash
computation takes a few parameters into account.

How hash keys are computed. Copy Chapter

Rust Copy Section

We generate a blake3 digest for each file compiled.
In parallel, we also take into account in the hash:

• Path to the rustc executable
• Host triple for this rustc
• Path to the rustc sysroot
• digests of all the shared libraries in rustc's $sysroot/lib
• A shared, caching reader for rlib dependencies (for dist-client)
• Parsed arguments from the rustc invocation

See https://github.com/mozilla/sccache/blob/8567bbe2ba493153e76177c1f9a6f98cc7ba419f/src/compiler/rust.rs#L122 for the full list

C/C++ compiler Copy Section

For C/C++, the hash is generated with a blake3 digest of the preprocessed
file (-E with gcc/clang). For compilations that specify multiple -arch flags,
these flags are rewritten to their corresponding preprocessor defines to allow
pre-processing the file (e.g -arch x86_64 is rewritten to -D__X86_64__=1),
this can be enabled by setting the environment variable
SCCACHE_CACHE_MULTIARCH but is disabled by default as it may not work in all
cases.

We also take into account in the hash:

• Hash of the compiler binary
• Programming language
• Flag required to compile for the given language
• File in which to generate dependencies.
• Commandline arguments for dependency generation
• Commandline arguments for the preprocessor
• Commandline arguments specifying the architecture to compile for
• Extra files that need to have their contents hashed
• Whether the compilation is generating profiling or coverage data
• Color mode
• Environment variables

See https://github.com/mozilla/sccache/blob/8567bbe2ba493153e76177c1f9a6f98cc7ba419f/src/compiler/c.rs#L84

C/C++ preprocessor Copy Section

In "preprocessor cache mode", explained in the local doc, an
extra key is computed to cache the preprocessor output itself. It is very close
to the C/C++ compiler one, but with additional elements:

• The path of the input file
• The hash of the input file

Note that some compiler options can disable preprocessor cache mode. As of this
writing, only -Xpreprocessor and -Wp,* do.

File: docs/Configuration.md Copy Chapter

Available Configuration Options

file Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

sccache looks for its configuration file at the path indicated by env variable SCCACHE_CONF.

If no such env variable is set, sccache looks at default locations as below:

• Linux: ~/.config/sccache/config
• macOS: ~/Library/Application Support/Mozilla.sccache/config
• Windows: %APPDATA%\Mozilla\sccache\config\config

The latest cache.XXX entries may be found here: https://github.com/mozilla/sccache/blob/ffe3070f77ef3301c8ff718316e4ab017ec83042/src/config.rs#L300.

env Copy Chapter

Whatever is set by a file based configuration, it is overruled by the env
configuration variables

Note that some env variables may need sccache server restart to take effect.

misc Copy Section

• SCCACHE_ALLOW_CORE_DUMPS to enable core dumps by the server
• SCCACHE_CONF configuration file path
• SCCACHE_BASEDIRS base directory (or directories) to strip from paths for cache key computation. This is similar to ccache's CCACHE_BASEDIR and enables cache hits across different absolute paths when compiling the same source code. Multiple directories can be separated by ; on Windows hosts and by : on any other operating system. When multiple directories are specified, the longest matching prefix is used. Path matching is case-insensitive on Windows and case-sensitive on other operating systems. Environment variable takes precedence over file configuration. Only absolute paths are supported; relative paths will cause an error and prevent the server from start.
• SCCACHE_CACHED_CONF
• SCCACHE_IDLE_TIMEOUT how long the local daemon process waits for more client requests before exiting, in seconds. Set to 0 to run sccache permanently
• SCCACHE_STARTUP_NOTIFY specify a path to a socket which will be used for server completion notification
• SCCACHE_MAX_FRAME_LENGTH how much data can be transferred between client and server
• SCCACHE_NO_DAEMON set to 1 to disable putting the server to the background
• SCCACHE_CLIENT_SIDE set to 1 to run the compile in the client process and use the daemon only as a gateway to the cache storage (see the architecture doc). This is the recommended mode and is expected to become the only supported configuration in the future. Ignored when SCCACHE_ERROR_LOG or distributed compilation is in use.
• SCCACHE_CACHE_MULTIARCH to disable caching of multi architecture builds.
• SCCACHE_CACHE_ZSTD_LEVEL to set zstd compression level of cache. the range is 1-22 and default is 3.
  • For example, in 10, it have about 0.9x size with about 1.6x time than default 3 (tested with compiling sccache code)
  • This option will only applied to newly compressed cache and don't affect existing cache.
  • If you want to be apply to all cache, you should reset cache and make new cache.
• SCCACHE_LOG_MILLIS when set (to any value), enables millisecond precision timestamps in log output instead of the default second precision.
• SCCACHE_ERROR_LOG path to a file where sccache will log errors
• SCCACHE_LOG log level, accepting standard env_logger values, see env_logger documentation for details

cache configs Copy Section

multi-level cache

Multi-level caching enables hierarchical cache storage with automatic backfill. See the Multi-Level Cache documentation for detailed information.

• SCCACHE_MULTILEVEL_CHAIN comma-separated list of cache backend names to use in hierarchy (e.g., disk,redis,s3)
  • Order matters: left-to-right is fast-to-slow (L0, L1, L2, ...)
  • Valid names: disk, redis, memcached, s3, gcs, azure, gha, webdav, oss, cos
  • Each level must be separately configured with its own environment variables
  • If not set, sccache uses single-level mode (legacy behavior)
• SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY controls error handling on cache writes (default: l0)
  • ignore - never fail on write errors, log warnings only (most permissive)
  • l0 - fail only if L0 (first level) write fails (default, balances reliability and performance)
  • all - fail if any read-write level fails (most strict)
  • Read-only levels are always skipped and never cause failures

Basic example:

export SCCACHE_MULTILEVEL_CHAIN="disk,redis,s3"
export SCCACHE_DIR="/tmp/cache"              # for disk level
export SCCACHE_REDIS_ENDPOINT="redis://..."  # for redis level
export SCCACHE_BUCKET="my-bucket"            # for s3 level

Write policy examples:

# Default: Fail only if disk write fails
export SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY="l0"

# Best effort: Never fail on cache writes
export SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY="ignore"

# Strict: Fail if any level write fails
export SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY="all"

disk (local)

• SCCACHE_DIR local on disk artifact cache directory
• SCCACHE_CACHE_SIZE maximum size of the local on disk cache i.e. 2G - default is 10G
• SCCACHE_DIRECT enable/disable preprocessor caching (see the local doc)
• SCCACHE_LOCAL_RW_MODE the mode that the cache will operate in (READ_ONLY or READ_WRITE)

s3 compatible

• SCCACHE_BUCKET s3 bucket to be used
• SCCACHE_ENDPOINT s3 endpoint
• SCCACHE_REGION s3 region, required if using AWS S3
• SCCACHE_S3_USE_SSL s3 endpoint requires TLS, set this to true
• SCCACHE_S3_KEY_PREFIX s3 key prefix (optional)
• SCCACHE_S3_RW_MODE allows to use s3 backend in read-only mode if set to READ_ONLY

The endpoint used then becomes ${SCCACHE_BUCKET}.s3-{SCCACHE_REGION}.amazonaws.com.
If you are not using the default endpoint and SCCACHE_REGION is undefined, it
will default to us-east-1.

cloudflare r2

• SCCACHE_BUCKET is the name of your R2 bucket.
• SCCACHE_ENDPOINT must follow the format of https://<ACCOUNT_ID>.r2.cloudflarestorage.com. Note that the https:// must be included. Your account ID can be found here.
• SCCACHE_REGION should be set to auto.
• SCCACHE_S3_KEY_PREFIX s3 key prefix (optional).

redis

• SCCACHE_REDIS full redis url, including auth and access token/passwd (deprecated).
• SCCACHE_REDIS_ENDPOINT redis url without auth and access token/passwd - single node configuration.
• SCCACHE_REDIS_CLUSTER_ENDPOINTS redis cluster urls, separated by comma - shared cluster configuration.
• SCCACHE_REDIS_USERNAME redis username (optional).
• SCCACHE_REDIS_PASSWORD redis password (optional).
• SCCACHE_REDIS_DB redis database (optional, default is 0).
• SCCACHE_REDIS_EXPIRATION / SCCACHE_REDIS_TTL ttl for redis cache, don't set for default behavior.
• SCCACHE_REDIS_KEY_PREFIX key prefix (optional).
• SCCACHE_REDIS_RW_MODE allows to use redis backend in read-only mode if set to READ_ONLY

The full url appears then as redis://user:[email protected]:6379/?db=1.

memcached

• SCCACHE_MEMCACHED is a deprecated alias for SCCACHE_MEMCACHED_ENDPOINT.
• SCCACHE_MEMCACHED_ENDPOINT memcached url.
• SCCACHE_MEMCACHED_USERNAME memcached username (optional).
• SCCACHE_MEMCACHED_PASSWORD memcached password (optional).
• SCCACHE_MEMCACHED_EXPIRATION ttl for memcached cache, don't set for default behavior.
• SCCACHE_MEMCACHED_KEY_PREFIX key prefix (optional).
• SCCACHE_MEMCACHED_RW_MODE allows to use memcached backend in read-only mode if set to READ_ONLY

gcs

• SCCACHE_GCS_BUCKET
• SCCACHE_GCS_CREDENTIALS_URL
• SCCACHE_GCS_KEY_PATH
• SCCACHE_GCS_RW_MODE

azure

• SCCACHE_AZURE_BLOB_CONTAINER
• SCCACHE_AZURE_CONNECTION_STRING shared-key connection string. Mutually exclusive with the Entra ID variables below.
• SCCACHE_AZURE_STORAGE_ACCOUNT storage account name for Microsoft Entra ID (passwordless) auth; the endpoint https://{account}.blob.core.windows.net is synthesized.
• SCCACHE_AZURE_ENDPOINT full blob endpoint override for Entra ID auth (sovereign clouds, custom DNS); must be https (plain http only for a loopback host); takes precedence over SCCACHE_AZURE_STORAGE_ACCOUNT.
• SCCACHE_AZURE_KEY_PREFIX
• SCCACHE_AZURE_RW_MODE

When using Entra ID, credentials are read from the ambient AZURE_* environment (AZURE_TENANT_ID / AZURE_CLIENT_ID / AZURE_CLIENT_SECRET for a service principal, AZURE_FEDERATED_TOKEN_FILE for workload identity, or IMDS for a managed identity). See Azure.md.

gha

• SCCACHE_GHA_CACHE_URL / ACTIONS_RESULTS_URL GitHub Actions cache API URL
• SCCACHE_GHA_RUNTIME_TOKEN / ACTIONS_RUNTIME_TOKEN GitHub Actions access token
• SCCACHE_GHA_CACHE_TO cache key to write
• SCCACHE_GHA_CACHE_FROM comma separated list of cache keys to read from
• SCCACHE_GHA_RW_MODE allows to use GHA cache backend in read-only mode if set to READ_ONLY

webdav

• SCCACHE_WEBDAV_ENDPOINT a webdav service endpoint to store cache, such as http://127.0.0.1:8080/my/webdav.php.
• SCCACHE_WEBDAV_KEY_PREFIX specify the key prefix (subfolder) of cache (optional).
• SCCACHE_WEBDAV_USERNAME a username to authenticate with webdav service (optional).
• SCCACHE_WEBDAV_PASSWORD a password to authenticate with webdav service (optional).
• SCCACHE_WEBDAV_TOKEN a token to authenticate with webdav service (optional) - may be used instead of login & password.
• SCCACHE_WEBDAV_RW_MODE allows to use webdav backend in read-only mode if set to READ_ONLY

OSS

• SCCACHE_OSS_BUCKET
• SCCACHE_OSS_ENDPOINT
• SCCACHE_OSS_KEY_PREFIX
• ALIBABA_CLOUD_ACCESS_KEY_ID
• ALIBABA_CLOUD_ACCESS_KEY_SECRET
• SCCACHE_OSS_NO_CREDENTIALS
• SCCACHE_OSS_RW_MODE

Tencent Cloud Object Storage (COS)

• SCCACHE_COS_BUCKET
• SCCACHE_COS_ENDPOINT
• SCCACHE_COS_KEY_PREFIX
• TENCENTCLOUD_SECRET_ID
• TENCENTCLOUD_SECRET_KEY
• SCCACHE_COS_RW_MODE

File: docs/Distributed.md Copy Chapter

Distributed sccache

Background:

• You should read about JSON Web Tokens - https://jwt.io/.
  • HS256 in short: you can sign a piece of (typically unencrypted)
    data with a key. Verification involves signing the data again
    with the same key and comparing the result. As a result, if you
    want two parties to verify each others messages, the key must be
    shared beforehand.
• Secure token's referenced below should be generated with a CSPRNG
  (your OS random number generator should suffice).
  For example, on Linux this is accessible with: openssl rand -hex 64.
• When relying on random number generators (for generating keys or
  tokens), be aware that a lack of entropy is possible in cloud or
  virtualized environments in some scenarios.

Overview Copy Chapter

Distributed sccache consists of three parts:

• the client, an sccache binary that wishes to perform a compilation on
  remote machines
• the scheduler (sccache-dist binary), responsible for deciding where
  a compilation job should run
• the server (sccache-dist binary), responsible for actually executing
  a build

All servers are required to be a 64-bit Linux or a FreeBSD install. Clients
may request compilation from Linux, Windows or macOS. Linux compilations will
attempt to automatically package the compiler in use, while Windows and macOS
users will need to specify a toolchain for cross-compilation ahead of time.

Communication Copy Chapter

The HTTP implementation of sccache has the following API, where all HTTP body content is encoded using bincode:

• scheduler
  • POST /api/v1/scheduler/alloc_job
    • Called by a client to submit a compilation request.
    • Returns information on where the job is allocated it should run.
  • GET /api/v1/scheduler/server_certificate
    • Called by a client to retrieve the (dynamically created) HTTPS
      certificate for a server, for use in communication with that server.
    • Returns a digest and PEM for the temporary server HTTPS certificate.
  • POST /api/v1/scheduler/heartbeat_server
    • Called (repeatedly) by servers to register as available for jobs.
  • POST /api/v1/scheduler/job_state
    • Called by servers to inform the scheduler of the state of the job.
  • GET /api/v1/scheduler/status
    • Returns information about the scheduler.
• server
  • POST /api/v1/distserver/assign_job
    • Called by the scheduler to inform of a new job being assigned to this server.
    • Returns whether the toolchain is already on the server or needs submitting.
  • POST /api/v1/distserver/submit_toolchain
    • Called by the client to submit a toolchain.
  • POST /api/v1/distserver/run_job
    • Called by the client to run a job.
    • Returns the compilation stdout along with files created.

There are three axes of security in this setup:

1. Can the scheduler trust the servers?
2. Is the client permitted to submit and run jobs?
3. Can third parties see and/or modify traffic?

Server Trust Copy Section

If a server is malicious, they can return malicious compilation output to a user.
To protect against this, servers must be authenticated to the scheduler. You have three
means for doing this, and the scheduler and all servers must use the same mechanism.

Once a server has registered itself using the selected authentication, the scheduler
will trust the registered server address and use it for builds.

JWT HS256 (preferred)

This method uses secret key to create a per-IP-and-port token for each server.
Acquiring a token will only allow participation as a server if the attacker can
additionally impersonate the IP and port the token was generated for.

You must keep the secret key safe.

To use it:

Create a scheduler key with sccache-dist auth generate-jwt-hs256-key (which will
use your OS random number generator) and put it in your scheduler config file as
follows:

server_auth = { type = "jwt_hs256", secret_key = "YOUR_KEY_HERE" }

Now generate a token for the server, giving the IP and port the scheduler and clients can
connect to the server on (address 192.168.1.10:10501 here):

sccache-dist auth generate-jwt-hs256-server-token \
    --secret-key YOUR_KEY_HERE \
    --server 192.168.1.10:10501

or:

sccache-dist auth generate-jwt-hs256-server-token \
    --config /path/to/scheduler-config.toml \
    --server 192.168.1.10:10501

This will output a token (you can examine it with https://jwt.io if you're
curious) that you should add to your server config file as follows:

scheduler_auth = { type = "jwt_token", token = "YOUR_TOKEN_HERE" }

Done!

Token

This method simply shares a token between the scheduler and all servers. A token
leak from anywhere allows any attacker to participate as a server.

To use it:

Choose a 'secure token' you can share between your scheduler and all servers. The token must be a valid HTTP header value.

Put the following in your scheduler config file:

server_auth = { type = "token", token = "YOUR_TOKEN_HERE" }

Put the following in your server config file:

scheduler_auth = { type = "token", token = "YOUR_TOKEN_HERE" }

Done!

Insecure (bad idea)

This route is not recommended

This method uses a hardcoded token that effectively disables authentication and
provides no security at all.

To use it:

Put the following in your scheduler config file:

server_auth = { type = "DANGEROUSLY_INSECURE" }

Put the following in your server config file:

scheduler_auth = { type = "DANGEROUSLY_INSECURE" }

Done!

Client Trust Copy Section

If a client is malicious, they can cause a DoS of distributed sccache servers or
explore ways to escape the build sandbox. To protect against this, clients must
be authenticated.

Each client will use an authentication token for the initial job allocation request
to the scheduler. A successful allocation will return a job token that is used
to authorise requests to the appropriate server for that specific job.

This job token is a JWT HS256 token of the job id, signed with a server key.
The key for each server is randomly generated on server startup and given to
the scheduler during registration. This means that the server can verify users
without either a) adding client authentication to every server or b) needing
secret transfer between scheduler and server on every job allocation.

OAuth2

This is a group of similar methods for achieving the same thing - the client
retrieves a token from an OAuth2 service, and then submits it to the scheduler
which has a few different options for performing validation on that token.

To use it:

Put one of the following settings in your scheduler config file to determine how
the scheduler will validate tokens from the client:

# Use the known settings for Mozilla OAuth2 token validation
client_auth = { type = "mozilla" }

# Will forward the valid JWT token onto another URL in the `Bearer` header, with a
# success response indicating the token is valid. Optional `cache_secs` how long
# to cache successful authentication for.
client_auth = { type = "proxy_token", url = "...", cache_secs = 60 }

Additionally, each client should set up an OAuth2 configuration in the with one of
the following settings (as appropriate for your OAuth service):

# Use the known settings for Mozilla OAuth2 authentication
auth = { type = "mozilla" }

# Use the Authorization Code with PKCE flow. This requires a client id,
# an initial authorize URL (which may have parameters like 'audience' depending
# on your service) and the URL for retrieving a token after the browser flow.
auth = { type = "oauth2_code_grant_pkce", client_id = "...", auth_url = "...", token_url = "..." }

# Use the Implicit flow (typically not recommended due to security issues). This requires
# a client id and an authorize URL (which may have parameters like 'audience' depending
# on your service).
auth = { type = "oauth2_implicit", client_id = "...", auth_url = "..." }

The client should then run sccache --dist-auth and follow the instructions to retrieve
a token. This will be automatically cached locally for the token expiry period (manual
revalidation will be necessary after expiry).

Token

This method simply shares a token between the scheduler and all clients. A token
leak from anywhere allows any attacker to participate as a client.

To use it:

Choose a 'secure token' you can share between your scheduler and all clients. The token must be a valid HTTP header value.

Put the following in your scheduler config file:

client_auth = { type = "token", token = "YOUR_TOKEN_HERE" }

Put the following in your client config file:

auth = { type = "token", token = "YOUR_TOKEN_HERE" }

Done!

Insecure (bad idea)

This route is not recommended

This method uses a hardcoded token that effectively disables authentication and
provides no security at all.

To use it:

Put the following in your scheduler config file:

client_auth = { type = "DANGEROUSLY_INSECURE" }

Remove any auth = setting under the [dist] heading in your client config file
(it will default to this insecure mode).

Done!

Eavesdropping and Tampering Protection Copy Section

If third parties can see traffic to the servers, source code can be leaked. If third
parties can modify traffic to and from the servers or the scheduler, they can cause
the client to receive malicious compiled objects.

Securing communication with the scheduler is the responsibility of the sccache cluster
administrator - it is recommended to put a webserver with a HTTPS certificate in front
of the scheduler and instruct clients to configure their scheduler_url with the
appropriate https:// address. The scheduler will verify the server's IP in this
configuration by inspecting the X-Real-IP header's value, if present. The webserver
used in this case should be configured to set this header to the appropriate value.

Securing communication with the server is performed automatically - HTTPS certificates
are generated dynamically on server startup and communicated to the scheduler during
the heartbeat. If a client does not have the appropriate certificate for communicating
securely with a server (after receiving a job allocation from the scheduler), the
certificate will be requested from the scheduler.

Configuration Copy Chapter

Use the --config argument to pass the path to its configuration file to sccache-dist.

scheduler.toml Copy Section

# The socket address the scheduler will listen on. It's strongly recommended
# to listen on localhost and put a HTTPS server in front of it.
public_addr = "127.0.0.1:10600"

[client_auth]
type = "token"
token = "my client token"

[server_auth]
type = "jwt_hs256"
secret_key = "my secret key"

[client_auth]

The [client_auth] section can be one of (sorted by authentication method):

# OAuth2
[client_auth]
type = "mozilla"

client_auth = { type = "proxy_token", url = "...", cache_secs = 60 }

# JWT
[client_auth]
type = "jwt_validate"
audience = "audience"
issuer = "issuer"
jwks_url = "..."

# Token
[client_auth]
type = "token"
token = "preshared token"

# None
[client_auth]
type = "DANGEROUSLY_INSECURE"

[server_auth]

The [server_auth] section can be can be one of:

[server_auth]
type = "jwt_hs256"
secret_key = "my secret key"

[server_auth]
type = "token"
token = "preshared token"

[server_auth]
type = "DANGEROUSLY_INSECURE"

server.toml Copy Section

# This is where client toolchains will be stored.
cache_dir = "/tmp/toolchains"
# The maximum size of the toolchain cache, in bytes.
# If unspecified the default is 10GB.
#toolchain_cache_size = 10737418240
# A public IP address and port that clients will use to connect to this builder.
public_addr = "192.168.1.1:10501"
# The socket address the builder will listen on. Falls back to public_addr.
#bind_address = "0.0.0.0:10501"
# The URL used to connect to the scheduler (should use https, given an ideal
# setup of a HTTPS server in front of the scheduler)
scheduler_url = "https://192.168.1.1"

[builder]
type = "overlay"
# The directory under which a sandboxed filesystem will be created for builds.
build_dir = "/tmp/build"
# The path to the bubblewrap version 0.3.0+ `bwrap` binary.
bwrap_path = "/usr/bin/bwrap"

[scheduler_auth]
type = "jwt_token"
# This will be generated by the `generate-jwt-hs256-server-token` command or
# provided by an administrator of the sccache cluster.
token = "my server's token"

[builder]

The [builder] section can be can be one of:

[builder]
type = "docker"

[builder]
type = "overlay"
# The directory under which a sandboxed filesystem will be created for builds.
build_dir = "/tmp/build"
# The path to the bubblewrap version 0.3.0+ `bwrap` binary.
bwrap_path = "/usr/bin/bwrap"

[builder]
type = "pot"
# Pot filesystem root
#pot_fs_root = "/opt/pot"
# Reference pot cloned when creating containers
#clone_from = "sccache-template"
# Command to invoke when calling pot
#pot_cmd = "pot"
# Arguments passed to `pot clone` command
#pot_clone_args = ["-i", "lo0|127.0.0.2"]

[scheduler_auth]

The [scheduler_auth] section can be can be one of:

[scheduler_auth]
type = "jwt_token"
token = "my server's token"

[scheduler_auth]
type = "token"
token = "preshared token"

[scheduler_auth]
type = "DANGEROUSLY_INSECURE"

Building the Distributed Server Binaries

Until these binaries are included in releases I've put together a Docker container that can be used to easily build a release binary:

docker run -ti --rm -v $PWD:/sccache luser/sccache-musl-build:0.1 /bin/bash -c "cd /sccache; cargo build --release --target x86_64-unknown-linux-musl --features=dist-server && strip target/x86_64-unknown-linux-musl/release/sccache-dist && cd target/x86_64-unknown-linux-musl/release/ && tar czf sccache-dist.tar.gz sccache-dist"

File: docs/DistributedFreeBSD.md Copy Chapter

Distributed sccache on FreeBSD

Please read the the distributed quickstart
guide first.

Build and install from source Copy Chapter

cargo install --features="dist-client,dist-server" --path=.

Configure a FreeBSD build server Copy Chapter

On FreeBSD, the build server requires pot
to sandbox execution:

pkg install pot

It's up to the user to create the reference pot that serves as a template
to clone from when instantiating image and build containers, e.g.:

pot create -p sccache-template -N alias -i "lo0|127.0.0.2" -t single -b 15.0
pot set-cmd -p sccache-template -c /usr/bin/true
pot set-attr -p sccache-template -A no-rc-script -V YES
pot snapshot -p sccache-template

Then, a server.conf like the one below is created, making use of the pot
builder type (commented out options show defaults):

# This is where client toolchains will be stored.
cache_dir = "/tmp/toolchains"
# The maximum size of the toolchain cache, in bytes.
# If unspecified the default is 10GB.
# toolchain_cache_size = 10737418240
# A public IP address and port that clients will use to connect to this builder.
public_addr = "192.168.1.1:10501"
# The URL used to connect to the scheduler (should use https, given an ideal
# setup of a HTTPS server in front of the scheduler)
scheduler_url = "https://192.168.1.1"

[builder]
type = "pot"
# Pot filesystem root
#pot_fs_root = "/opt/pot"
# Reference pot cloned when creating containers
#clone_from = "sccache-template"
# Command to invoke when calling pot
#pot_cmd = "pot"
# Arguments passed to `pot clone` command
#pot_clone_args = ["-i", "lo0|127.0.0.2"]

[scheduler_auth]
type = "jwt_token"
# This will be generated by calling
# `sccache-dist auth generate-jwt-hs256-server-token` or
# provided by an administrator of the sccache cluster.
token = "my server's token"

FreeBSD as a build client Copy Chapter

On a FreeBSD client, make sure to add the right toolchains to
~/.config/sccache/config:

[dist]
# The URL used to connect to the scheduler (should use https, given an ideal
# setup of a HTTPS server in front of the scheduler)
scheduler_url = "http://127.0.0.1:10600"
# Used for mapping local toolchains to remote cross-compile toolchains. Empty in
# this example where the client and build server are both Linux.
#toolchains = []
# Size of the local toolchain cache, in bytes (5GB here, 10GB if unspecified).
toolchain_cache_size = 5368709120
cache_dir = "/home/user/.cache/sccache-dist-client"

[dist.auth]
type = "token"
# This should match the `client_auth` section of the scheduler config
# and was generated by calling `sccache-dist auth generate-jwt-hs256-key`
token = "my client token"

[[dist.toolchains]]
type = "path_override"
compiler_executable = "/usr/bin/cc"
archive = "/path/to/empty.tar.gz"
archive_compiler_executable = "/usr/bin/cc"

[[dist.toolchains]]
type = "path_override"
compiler_executable = "/usr/local/bin/rustc"
archive = "/path/to/rust-toolchain.tgz"
archive_compiler_executable = "/usr/local/bin/rustc"

Creating toolchain archives Copy Chapter

The toolchain files from the examples above can be created like this:

pkg install gtar
gtar cvf - --files-from /dev/null | gzip >empty.tar.gz
pkg info -lq rust | gtar -cf - -T - | gzip >rust-toolchain.tgz

This just creates an empty file for the system compiler (as it is
included in the pot image anyway) and the toolchain for rustc is
created from the rust package installed on the system.

See the distributed quickstart guide for
instructions how to create other C toolchains using icecc-create-env.

Note: We use gtar (GNU tar) here, as the flate2 crate has issues processing
sparse files created with bsdtar.

Cargo invocation example Copy Chapter

RUSTC_WRAPPER=~/.cargo/bin/sccache \
cargo build

File: docs/DistributedQuickstart.md Copy Chapter

sccache distributed compilation quickstart

This is a quick start guide to getting distributed compilation working with sccache. This guide primarily covers Linux clients.
macOS and Windows clients are supported but have seen significantly less testing.

Get sccache binaries Copy Chapter

Either install pre-built sccache binaries, or build sccache locally with the dist-client and dist-server features enabled:

cargo build --release --features="dist-client dist-server"

The target/release/sccache binary will be used on the client, and the target/release/sccache-dist binary will be used on the scheduler and build server.

If you're only planning to use the client, it is enabled by default, so just cargo install sccache should do the trick.

Configure a scheduler Copy Chapter

If you're adding a server to a cluster that has already been set up, skip ahead to configuring a build server.

The scheduler is a daemon that manages compile request from clients and parcels them out to build servers. You only need one of these per sccache setup. Currently only Linux is supported for running the scheduler.

Create a scheduler.conf file to configure client/server authentication. A minimal example looks like:

# The socket address the scheduler will listen on. It's strongly recommended
# to listen on localhost and put a HTTPS server in front of it.
public_addr = "127.0.0.1:10600"

[client_auth]
type = "token"
token = "my client token" # Must be a valid HTTP header value.

[server_auth]
type = "jwt_hs256"
secret_key = "my secret key"

Mozilla build servers will typically require clients to be authenticated with the
Mozilla identity system.

To configure for scheduler for this, the client_auth section should be as follows
so any client tokens are validated with the Mozilla service:

[client_auth]
type = "mozilla"
required_groups = ["group_name"]

Where group_name is a Mozilla LDAP group. Users will be required to belong to this group to successfully authenticate with the scheduler.

Start the scheduler by running:

sccache-dist scheduler --config scheduler.conf

Like the local server, the scheduler process will daemonize itself unless SCCACHE_NO_DAEMON=1 is set. If the scheduler fails to start you may need to set SCCACHE_LOG=trace when starting it to get useful diagnostics.

Configure a build server Copy Chapter

A build server communicates with the scheduler and executes compiles requested by clients. Only Linux is supported for running a build server, but executing cross-compile requests from macOS/Windows clients is supported. You can also run a build server on FreeBSD, please see distributed sccache on FreeBSD.

The build server requires bubblewrap to sandbox execution, at least version 0.3.0. Verify your version of bubblewrap before attempting to run the server. On Ubuntu 18.10+ you can apt install bubblewrap to install it. If you build from source you will need to first install your distro's equivalent of the libcap-dev package.

Create a server.conf file to configure authentication, storage locations, network addresses and the path to bubblewrap. A minimal example looks like:

# This is where client toolchains will be stored.
cache_dir = "/tmp/toolchains"
# The maximum size of the toolchain cache, in bytes.
# If unspecified the default is 10GB.
# toolchain_cache_size = 10737418240
# A public IP address and port that clients will use to connect to this builder.
public_addr = "192.168.1.1:10501"
# The URL used to connect to the scheduler (should use https, given an ideal
# setup of a HTTPS server in front of the scheduler)
scheduler_url = "https://192.168.1.1"

[builder]
type = "overlay"
# The directory under which a sandboxed filesystem will be created for builds.
build_dir = "/tmp/build"
# The path to the bubblewrap version 0.3.0+ `bwrap` binary.
bwrap_path = "/usr/bin/bwrap"

[scheduler_auth]
type = "jwt_token"
# This will be generated by the `generate-jwt-hs256-server-token` command or
# provided by an administrator of the sccache cluster.
token = "my server's token"

Due to bubblewrap requirements currently the build server must be run as root. Start the build server by running:

sudo sccache-dist server --config server.conf

As with the scheduler, if the build server fails to start you may need to set SCCACHE_LOG=trace to get useful diagnostics.

Configure a client Copy Chapter

A client uses sccache to wrap compile commands, communicates with the scheduler to find available build servers, and communicates with build servers to execute the compiles and receive the results.

Clients that are not targeting linux64 require the icecc-create-env script or should be provided with an archive. icecc-create-env is part of icecream for packaging toolchains. You can install icecream to get this script (apt install icecc on Ubuntu), or download it from the git repository and place it in your PATH: curl https://raw.githubusercontent.com/icecc/icecream/master/client/icecc-create-env.in > icecc-create-env && chmod +x icecc-create-env. See using custom toolchains.

Create a client config file in ~/.config/sccache/config (on Linux), ~/Library/Application Support/Mozilla.sccache/config (on macOS), or %APPDATA%\Mozilla\sccache\config\config (on Windows). A minimal example looks like:

[dist]
# The URL used to connect to the scheduler (should use https, given an ideal
# setup of a HTTPS server in front of the scheduler)
scheduler_url = "https://192.168.1.1"
# Used for mapping local toolchains to remote cross-compile toolchains. Empty in
# this example where the client and build server are both Linux.
toolchains = []
# Size of the local toolchain cache, in bytes (5GB here, 10GB if unspecified).
toolchain_cache_size = 5368709120

[dist.auth]
type = "token"
# This should match the `client_auth` section of the scheduler config.
token = "my client token" # Must be a valid HTTP header value.

Clients using Mozilla build servers should configure their dist.auth section as follows:

[dist.auth]
type = "mozilla"

And retrieve a token from the Mozilla identity service by running sccache --dist-auth
and following the instructions. Completing this process will retrieve and cache a token
valid for 7 days.

Make sure to run sccache --stop-server and sccache --start-server if sccache was
running before changing the configuration.

You can check the status with sccache --dist-status, it should say something like:

$ sccache --dist-status
{"SchedulerStatus":["https://sccache1.corpdmz.ber3.mozilla.com/",{"num_servers":3,"num_cpus":56,"in_progress":24}]}

Using custom toolchains Copy Chapter

Since Windows and macOS cannot automatically package toolchains, it is important to be
able to manually specify toolchains for distribution. This functionality is also available
on Linux.

Using custom toolchains involves adding a dist.toolchains section to your client config
file (you can add it multiple times to specify multiple toolchains).

On Linux and macOS:

[[dist.toolchains]]
type = "path_override"
compiler_executable = "/home/me/.mozbuild/clang/bin/clang"
archive = "/home/me/.mozbuild/toolchains/33d92fcd79ffef6e-clang-dist-toolchain.tar.gz"
archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"

On Windows:

[[dist.toolchains]]
type = "path_override"
compiler_executable = "C:/clang/bin\\clang-cl.exe"
archive = "C:/toolchains/33d92fcd79ffef6e-clang-dist-toolchain.tar.gz"
archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"

Where:

• compiler_executable identifies the path that sccache will match against to activate
  this configuration (you need to be careful on Windows - paths can have slashes in both
  directions, and you may need to escape backslashes, as in the example)
• archive is the gzip-compressed tar archive containing the compiler toolchain to distribute
  when compiler_executable is matched
• archive_compiler_executable is the path within the archive the distributed
  compilation should invoke

A toolchain archive should be a Gzip compressed TAR archive, containing a filesystem
sufficient to run the compiler without relying on any external files. If you have archives
compatible with icecream (created with icecc-create-env, like
these ones for macOS), they should also work
with sccache. To create a Windows toolchain, it is recommended that you download the Clang
binaries for Ubuntu 16.04 and extract them,
package up the toolchain using the extracted bin/clang file (requires
PR #321) and then insert bin/clang-cl at
the appropriate path as a symlink to the bin/clang binary.

Considerations when distributing from macOS Copy Chapter

When distributing from a macOS client, additional flags and configuration
may be required:

• An explicit target should be passed to the compiler, for instance by adding
  --target=x86_64-apple-darwin16.0.0 to your build system's CFLAGS.
• An explicit toolchain archive will need to be configured, as described above.
  In case rust is being cached, the same version of rustc will need to be used
  for local compiles as is found in the distributed archive.
• The client config will be read from ~/Library/Application Support/Mozilla.sccache/config,
  not ~/.config/sccache/config.
• Some cross compilers may not understand some intrinsics used in more recent macOS
  SDKs. The 10.11 SDK is known to work.

Making a build server start at boot time Copy Chapter

It is very easy with a systemd service to spawn the server on boot.

You can create a service file like /etc/systemd/system/sccache-server.service
with the following contents:

[Unit]
Description=sccache-dist server
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/path/to/sccache-dist server --config /path/to/server.conf

[Install]
WantedBy=multi-user.target

Note that if the sccache-dist binary is in a user's home directory, and
you're in a distro with SELinux enabled (like Fedora), you may need to use an
ExecStart line like:

ExecStart=/bin/bash -c "/home//path/to/sccache-dist server --config /home//path/to/server.conf"

This is because SELinux by default prevents services from running binaries in
home directories, for some reason. Using a shell works around that. An
alternative would be to move the sccache-dist binary to somewhere like
/usr/local/bin, but then you need to remember to update it manually.

After creating that file, you can ensure it's working and enable it by default
like:

# systemctl daemon-reload
# systemctl start sccache-server
# systemctl status # And check it's fine.
# systemctl enable sccache-server # This enables the service on boot

File: docs/Gcs.md Copy Chapter

Google Cloud Storage

To use Google Cloud Storage, you need to set
the SCCACHE_GCS_BUCKET environment variable to the name of the GCS bucket.

By default, SCCACHE on GCS will be read-only. To change this, set SCCACHE_GCS_RW_MODE
to either READ_ONLY or READ_WRITE.

You can also define a prefix that will be prepended to the keys of all cache objects
created and read within the GCS bucket, effectively creating a scope. To do that
use the SCCACHE_GCS_KEY_PREFIX environment variable. This can be useful when
sharing a bucket with another application.

Credentials Copy Chapter

Sccache is able to load credentials from various sources. Including:

• User Input: If SCCACHE_GCS_KEY_PATH has been set, we will load from this file
  first.
  • Service accounts JSONs
  • External accounts JSONs
• Task Cluster: If SCCACHE_GCS_CREDENTIALS_URL has
  been set, we will load token from this url first.
• Static: GOOGLE_APPLICATION_CREDENTIALS
• Well-known locations:
  • Windows: %APPDATA%\gcloud\application_default_credentials.json
  • macOS/Linux:
    • $XDG_CONFIG_HOME/gcloud/application_default_credentials.json
    • $HOME/.config/gcloud/application_default_credentials.json
• VM Metadata: Fetch token will the specified service account.

Service accounts Copy Section

To create such account, in GCP, go in APIs and Services => Cloud Storage =>
Create credentials => Service account. Then, once created, click on the account
then Keys => Add key => Create new key. Select the JSON format and here it
is. This JSON file is what SCCACHE_GCS_KEY_PATH expects.

The service account needs Storage Object Admin permissions on the bucket
(otherwise, sccache will fail with a simple Permission denied).

External accounts Copy Section

Such accounts require creating a Workload Identity Pool and Workload Identity Provider.
This approach allows the environment (Azure, Aws, or other OIDC providers like Github)
to create a temporary service account grant without having to share a service account
JSON, which can be pretty powerful. An example on how to create such accounts is
Google's guide on how to use it with Github.

After generating the external account JSON file, you may pass its path to SCCACHE_GCS_KEY_PATH.

Service accounts used by the pool must have Storage Object Admin permissions on
bucket as well.

Verifying it works Copy Chapter

To verify that it works, run:

export SCCACHE_GCS_BUCKET=
export SCCACHE_GCS_KEY_PATH=secret-gcp-storage.json
./sccache --show-stats
# you should see
[...]
Cache location                  GCS, bucket: Bucket(name=), key_prefix: (none)

Deprecation Copy Chapter

SCCACHE_GCS_OAUTH_URL have been deprecated and not supported, please use SCCACHE_GCS_SERVICE_ACCOUNT instead.

File: docs/GHA.md Copy Chapter

GitHub Actions

To use the GitHub Actions cache, you need to set SCCACHE_GHA_ENABLED to on to enable it.

By changing SCCACHE_GHA_VERSION, we can purge all the cache.

This cache type will need tokens like ACTIONS_RESULTS_URL and ACTIONS_RUNTIME_TOKEN to work. You can set these environmental variables using the following step in a GitHub Actions workflow.

- name: Configure sccache
  uses: actions/github-script@v7
  with:
    script: |
      core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
      core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');

The SCCACHE_GHA_RW_MODE environment variable can be set to READ_ONLY to make sccache use this backend in read-only mode. The default is READ_WRITE.

Behavior Copy Chapter

In case sccache reaches the rate limit of the service, the build will continue, but the storage might not be performed.