## File: README.md

[](https://trendshift.io/repositories/11202)

## What's NativeLink?

NativeLink is an efficient, high-performance build cache and remote execution system that accelerates software compilation and testing while reducing infrastructure costs. It optimizes build processes for projects of all sizes by intelligently caching build artifacts and distributing tasks across multiple machines.

NativeLink is trusted in production environments to reduce costs and developer iteration times--handling over **billions of requests** per month for its customers, including large corporations such as **Samsung**.



## 🔑 Key Features

1. **Advanced Build Cache**:
   - Stores and reuses results of previous build steps for unchanged components
   - Significantly reduces build times, especially for incremental changes

2. **Efficient Remote Execution**:
   - Distributes build and test tasks across a network of machines
   - Parallelizes workloads for faster completion
   - Utilizes remote resources to offload computational burden from local machines
   - Ensures consistency with a uniform, controlled build environment

NativeLink seamlessly integrates with build tools that use the Remote Execution protocol, such as [Bazel](https://bazel.build), [Buck2](https://buck2.build), [Goma](https://chromium.googlesource.com/infra/goma/client/), and [Siso](https://chromium.googlesource.com/build/+/refs/heads/main/siso/README.md). CMake projects work too via [`recc`](https://buildgrid.gitlab.io/recc). See [Build CMake projects with NativeLink](https://docs.nativelink.com/getting-started/connect-your-build/cmake-recc). It supports Unix-based operating systems and Windows, ensuring broad compatibility across different development environments.

## 🚀 Quickstart

To start, you can deploy NativeLink as a Docker image (as shown below). The setups below are **production-grade** installations. See the [contribution docs](https://nativelink.com/docs/contribute/nix/) for instructions on how to build from source with [Bazel](https://nativelink.com/docs/contribute/bazel/), [Cargo](https://nativelink.com/docs/contribute/cargo/), and [Nix](https://nativelink.com/docs/contribute/nix/).

You can find a few example deployments in the [Docs](https://nativelink.com/docs/deployment-examples/kubernetes).

### 📦 Prebuilt images

Fast to spin up, but currently limited to `x86_64` systems. See the [container
registry](https://github.com/TraceMachina/nativelink/pkgs/container/nativelink)
for all image tags and the [contribution docs](https://nativelink.com/docs/contribute/nix)
for how to build the images yourself.

**Linux x86_64**

```bash
curl -O \
    https://raw.githubusercontent.com/TraceMachina/nativelink/v1.4.0/nativelink-config/examples/basic_cas.json5

# See https://github.com/TraceMachina/nativelink/pkgs/container/nativelink
# to find the latest tag
docker run \
    -v $(pwd)/basic_cas.json5:/config \
    -p 50051:50051 \
    ghcr.io/tracemachina/nativelink:v1.4.0 \
    config
```

**Windows x86_64**

```powershell
# Download the configuration file
Invoke-WebRequest `
    -Uri "https://raw.githubusercontent.com/TraceMachina/nativelink/v1.4.0/nativelink-config/examples/basic_cas.json5" `
    -OutFile "basic_cas.json5"

# Run the Docker container
# Note: Adjust the path if the script is not run from the directory containing basic_cas.json
docker run `
    -v ${PWD}/basic_cas.json5:/config `
    -p 50051:50051 `
    ghcr.io/tracemachina/nativelink:v1.4.0 `
    config
```

### ❄️ Raw executable with Nix

Slower, since it's built from source, but more flexible and supports MacOS.
Doesn't support native Windows, but works in WSL2.

Make sure your Nix version is recent and supports flakes. For instance, install
it via the [next-gen nix installer](https://github.com/NixOS/experimental-nix-installer).

> [!CAUTION]
> Executables built for MacOS are dynamically linked against libraries from Nix
> and won't work on systems that don't have these libraries present.

> [!TIP]
> **Common setup gotchas for Nix on macOS / Linux:**
> * **Active shell environment**: If the installer finishes but your shell doesn't recognize `nix` commands, you need to either restart your terminal session or source the daemon profile manually:
>   ```bash
>   . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
>   ```
> * **Enabling experimental features**: If you used the standard Nix installer and get an error saying `experimental Nix feature 'nix-command' is disabled`, enable them by creating or editing `~/.config/nix/nix.conf`:
>   ```text
>   experimental-features = nix-command flakes
>   ```
> * **Disk Space**: Unpacking and building compilers and dependencies requires significant storage. Make sure you have at least **15–20 GB of free space** on your system volume before running setup commands.

**Linux, MacOS, WSL2**

```bash
curl -O \
    https://raw.githubusercontent.com/TraceMachina/nativelink/main/nativelink-config/examples/basic_cas.json5

nix run github:TraceMachina/nativelink ./basic_cas.json5
```

See the [contribution docs](https://nativelink.com/docs/contribute/nix) for further information.

## 🤝 Contributing

Visit our [Contributing](https://github.com/tracemachina/nativelink/blob/main/CONTRIBUTING.md) guide to learn how to contribute to NativeLink. We welcome contributions from developers of all skill levels and backgrounds!

## 📊 Stats



## 📜 License

Copyright 2020–2026 Trace Machina, Inc.

Licensed under the Functional Source License, Version 1.1, Apache 2.0 Future License.
SPDX identifier: `FSL-1.1-Apache-2.0`.

After the second anniversary of the date this version was made available, you may use this
software under the Apache License, Version 2.0.

---

## File: deployment-examples/docker-compose/README.md

# Multi-Worker Docker Compose Deployment

This guide explains how to run NativeLink with multiple workers using Docker Compose for distributed build execution.

## Prerequisites

- **Architecture**: This setup supports **x86_64/amd64** and **ARM64** architectures, except for Apple Silicon, which is not supported by the test client container
- **Google Cloud CLI**: If using the test client container from `gcr.io/bazel-public/bazel`, you'll need:
  1. [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) installed
  2. Authentication via `gcloud auth login`
  3. Docker credential helper configured: `gcloud auth configure-docker`

  Alternatively, you can comment out the test-client service in the docker-compose file and run Bazel tests locally.

## Quick Start

0. Build the Docker container:

```sh
docker compose -f
      docker-compose-multi-worker.yml build
```

1. Start the multi-worker deployment:
```sh
docker compose -f docker-compose-multi-worker.yml up -d
```

2. Test with Bazel:
```sh
bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052 \
  --remote_default_exec_properties=cpu_count=2
```

## Configuration Requirements

### Critical: Shared CAS Storage

**All workers MUST share the same CAS storage path.** This is the most common configuration error.

**Important:** When starting workers, you'll see this validation warning:
```
Starting worker 'worker-1'. IMPORTANT: If running multiple workers, all workers
must share the same CAS storage path to avoid 'Object not found' errors.
```

✅ **Correct Configuration:**
```yaml
services:
  worker-1:
    volumes:
      - cas-data:/data/cas  # Shared volume
  worker-2:
    volumes:
      - cas-data:/data/cas  # Same shared volume
  worker-3:
    volumes:
      - cas-data:/data/cas  # Same shared volume

volumes:
  cas-data:  # Single shared volume for all workers
```

❌ **Incorrect Configuration (causes "Object not found" errors):**
```yaml
services:
  worker-1:
    volumes:
      - ./data/worker-1/cas:/data/cas  # Isolated path
  worker-2:
    volumes:
      - ./data/worker-2/cas:/data/cas  # Different isolated path
```

### Worker Configuration ([worker-shared-cas.json5](./worker-shared-cas.json5))

```json5
{
  stores: [
    {
      name: "SHARED_CAS",
      filesystem: {
        // All workers MUST use the same path
        content_path: "/data/cas/content",
        // Temp can be worker-specific
        temp_path: "/tmp/${WORKER_NAME}/temp",
        eviction_policy: {
          max_bytes: 10000000000,
        }
      }
    }
  ],
  workers: [
    {
      local: {
        name: "${WORKER_NAME}",
        cas_fast_slow_store: "SHARED_CAS",  // All workers use same store
        // Work directory should be worker-specific
        work_directory: "/tmp/${WORKER_NAME}/work",
        platform_properties: {
          cpu_count: { values: ["2"] },
          memory_gb: { values: ["2"] }
        }
      }
    }
  ]
}
```

## Configuration Files

### Multi-Worker Setup
- [`docker-compose-multi-worker.yml`](./docker-compose-multi-worker.yml) - Docker Compose file for multi-worker deployment
- [`test-multi-worker-simple.json5`](./test-multi-worker-simple.json5) - All-in-one configuration for testing multi-worker setup (validated working)
- [`cas-server-multi-worker.json5`](./cas-server-multi-worker.json5) - CAS server configuration for multi-worker deployment
- [`scheduler-multi-worker.json5`](./scheduler-multi-worker.json5) - Scheduler configuration for multi-worker deployment
- [`worker-shared-cas.json5`](./worker-shared-cas.json5) - Worker configuration template with shared CAS storage

### Single Worker Setup
- [`docker-compose.yml`](./docker-compose.yml) - Docker Compose file for single worker deployment
- [`local-storage-cas.json5`](./local-storage-cas.json5) - Local storage CAS configuration for single worker
- [`scheduler.json5`](./scheduler.json5) - Scheduler configuration for single worker deployment
- [`worker.json5`](./worker.json5) - Worker configuration for single worker deployment

## Scaling Workers

### Dynamic Scaling
```sh
# Scale to 5 workers
docker-compose -f docker-compose-multi-worker.yml up -d --scale worker=5

# Scale down to 2 workers
docker-compose -f docker-compose-multi-worker.yml up -d --scale worker=2
```

### Resource Limits
Each worker is configured with:
- CPU: 2 cores (configurable)
- Memory: 2GB (configurable)

Adjust in `docker-compose-multi-worker.yml`:
```yaml
deploy:
  resources:
    limits:
      cpus: '4'    # Increase to 4 cores
      memory: 4G   # Increase to 4GB
```

## Monitoring

### View Logs
```sh
# All workers
docker-compose -f docker-compose-multi-worker.yml logs -f

# Specific worker
docker-compose -f docker-compose-multi-worker.yml logs -f worker-1

# Scheduler
docker-compose -f docker-compose-multi-worker.yml logs -f scheduler
```

### Check Status
```sh
# View all containers
docker-compose -f docker-compose-multi-worker.yml ps

# Check CAS storage usage
docker exec -it $(docker-compose -f docker-compose-multi-worker.yml ps -q worker-1) \
  du -sh /data/cas/content
```

## Troubleshooting

### "Object not found" Errors
**Symptom:** Workers fail with enhanced error message:
```
Object 7fd25e01d12373a2d1712e446881c9246a9698da4e7eafecdaeeaaff62195a82-148
not found in either fast or slow store. If using multiple workers, ensure
all workers share the same CAS storage path.
```

**Cause:** Workers are using different CAS paths

**Solution:**
1. Check volume mounts: `docker inspect <worker-container> | grep -A 5 Mounts`
2. Ensure all workers mount the same `cas-data` volume
3. Verify `content_path` is identical in all worker configurations
4. Use the validated `test-multi-worker-simple.json5` as reference

### Workers Not Executing Jobs
**Symptom:** Jobs queue but workers remain idle

**Solution:**
1. Check scheduler connectivity: `docker logs scheduler | grep -i error`
2. Verify worker registration: `docker logs worker-1 | grep "Connected to scheduler"`
3. Check network: `docker network ls`

### Performance Issues
**Symptom:** Builds slower with multiple workers

**Solution:**
1. Check disk I/O: `docker stats`
2. Consider using SSD-backed storage
3. Increase cache size in `eviction_policy`
4. Monitor with: `docker-compose -f docker-compose-multi-worker.yml logs -f | grep -i slow`

## Testing Multi-Worker Setup

### Local Testing with All-in-One Configuration

#### Option 1: Using Docker (Recommended)
```sh
# Run nativelink using the Docker image you built
docker run --rm -it \
  -v $(pwd)/test-multi-worker-simple.json5:/config.json5 \
  -p 50051:50051 \
  -p 50052:50052 \
  nativelink:latest /config.json5

# In another terminal, test with Bazel
bazel build //:nativelink \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50051 \
  --remote_default_exec_properties=cpu_count=1 \
  --jobs=3
```

#### Option 2: Build and Run Locally (Advanced)
```sh
# Build nativelink locally first
cargo build --release

# Start multi-worker server (3 workers, shared CAS)
./target/release/nativelink test-multi-worker-simple.json5

# In another terminal, test with Bazel
bazel build //:nativelink \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50051 \
  --remote_default_exec_properties=cpu_count=1 \
  --jobs=3
```

### Basic Test
```sh
# Build a simple target
bazel build //src:hello_world \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052

# Check which worker executed
docker-compose -f docker-compose-multi-worker.yml logs | grep "Executing action"
```

### Stress Test
```sh
# Build entire project with parallel jobs
bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052 \
  --jobs=50  # High parallelism to distribute across workers
```

## Production Considerations

For production deployments:
1. Use persistent volumes backed by SSD
2. Implement health checks in docker-compose.yml
3. Use external storage (NFS, S3, etc.) for CAS
4. Monitor worker metrics
5. Set up log aggregation

See [Kubernetes deployment](../kubernetes/) for production-grade configurations.

---

## File: local-remote-execution/README.md

# Local Remote Execution

[](https://www.youtube.com/embed/uokjTev8myk?rel=0)

NativeLink's Local Remote Execution is a framework to build, distribute, and
rapidly iterate on custom toolchain setups that are transparent, fully hermetic,
and reproducible across machines of the same system architecture.

Local Remote Execution mirrors toolchains for remote execution in your local
development environment. This lets you reuse build artifacts with virtually
perfect cache hit rate across different repositories, developers, and CI.

> [!WARNING]
> LRE highly experimental and can be challenging to set up in downstream
> projects. Please reach out on the [NativeLink](https://join.slack.com/t/nativelink/shared_invite/zt-2forhp5n9-L7dTD21nCSY9_IRteQvZmw)
> Slack if you have any questions or ideas for improvement.

> [!NOTE]
> At the moment LRE only works on `x86_64-linux`.

## Pre-Requisites

- Nix 2.19.0 or later
- A functional local container setup

## ❄️ Flake-side setup

Your project must have a Nix flake and use [flake-parts](https://github.com/hercules-ci/flake-parts).

Add the `nativelink` repository to your flake inputs and make the `nixpkgs` from
`nativelink` the global input for your project:

```nix
inputs = {
  nixpkgs = {
    url = "github:nixos/nixpkgs";

    # IMPORTANT: This needs to follow the `nixpkgs` from nativelink so that
    # the local LRE toolchains are in sync with the remote toolchains.
    follows = "nativelink/nixpkgs";
  };
  nativelink = {
    # This repository provides the autogenerated LRE toolchains which are
    # dependent on the nixpkgs version in the nativelink repository. To keep the
    # local LRE toolchains aligned with remote LRE, we need to use the nixpkgs
    # used by nativelink as the the "global" nixpkgs. We do this by setting
    # `nixpkgs.follows = "nativelink/nixpkgs"` above.

    # Keep this commit in sync with the LRE commit that you import via Bazel.
    url = "github:TraceMachina/nativelink/64ed20a40964b8c606c7d65f76af840bcfc837fd";
  };
  # ...
};
```

Then, in your `flake-parts` imports, add the `nativelink` flake module:

```nix
imports = [
  inputs.nativelink.flakeModule
];
```

Finally, add the `lre.bazelrc` generator in your `devShell`'s `shellHook`:

```nix
devShells.default = pkgs.mkShell {
  # ...
  shellHook = ''
    # Generate lre.bazelrc which configures LRE toolchains when running
    # in the nix environment.
    ${config.lre.installationScript}
  '';
}
```

Now, when you enter the nix flake (automated via `direnv` or manually via
`nix develop`) it will generate a file called `lre.bazelrc` which contains
information about the LRE toolchain and configures Bazel. With the default
configuration it looks like this:

```bash
# These flags are dynamically generated by the LRE flake module.
#
# Add `try-import %workspace%/lre.bazelrc` to your .bazelrc to
# include these flags when running Bazel in a nix environment.

# These are the paths used by your local LRE config. If you get cache misses
# between local and remote execution, double-check these values against the
# toolchain configs in the `@local-remote-execution` repository at the
# commit that you imported in your `MODULE.bazel`.
#
# WARNING: No environment set. LRE will not work locally.

# Bazel-side configuration for LRE.
build --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build --define=EXECUTOR=remote
build --extra_execution_platforms=@local-remote-execution//generated-cc/config:platform
build --extra_toolchains=@local-remote-execution//generated-cc/config:cc-toolchain
```

In the snippet above you can see a warning that no local toolchain is
configured. LRE needs to know the remote toolchain configuration to make it
available locally. The `lre` settings take an `Env` input and
an optional `prefix` input to configure the generated `lre.bazelrc`:

```nix
# This is a top-level field, next to `packages` and `apps`, and `devShells`
lre = {
  # In this example we import the lre-cc environment from nativelink and make it
  # available locally.
  inherit (lre-cc.meta) Env;

  # This causes the `build` commands in the `lre.bazelrc` to be prefixed with
  # `lre` so that LRE is disabled by default and may be enabled by passing
  # `--config=lre` to a Bazel build.
  prefix = "lre";
};
```

Exit and re-enter the flake. The `lre.bazelrc` should look now like this:

```bash
# These flags are dynamically generated by the LRE flake module.
#
# Add `try-import %workspace%/lre.bazelrc` to your .bazelrc to
# include these flags when running Bazel in a nix environment.

# These are the paths used by your local LRE config. If you get cache misses
# between local and remote execution, double-check these values against the
# toolchain configs in the `@local-remote-execution` repository at the
# commit that you imported in your `MODULE.bazel`.
#
# PATH=/nix/store/lr2xhk791gy0lfwk4g5954kd8yfqsbsl-binutils-wrapper-2.41/bin:/nix/store/29xdc6ax5g41njf20h09gmf52qm1zws6-customClang/bin:/nix/store/kwlg5k0zmnkkj3p2kzi3b49sj73887j4-stdenv-linux/bin:/nix/store/asqa3kfq3maclk7cqqhrjvp7vriw6ahy-coreutils-9.5/bin:/nix/store/vf8cjkgwmgd4sb3vkxh6x9iar71s1w1c-findutils-4.9.0/bin:/nix/store/zs4ajc6bp81c92fj2l5r2d68ra0hmfbb-gnutar-1.35/bin
# CC=/nix/store/29xdc6ax5g41njf20h09gmf52qm1zws6-customClang/bin/customClang
# BAZEL_LINKOPTS=-L/nix/store/z56vdnayhj4v4s7g4rsg9b1jkpdblk91-libcxx-18.1.3/lib:-L/nix/store/1jvrjkymv56pbva6hfsbdavf8h44dn1g-libunwind-18.1.3/lib:-lc++:-Wl,-rpath,/nix/store/z56vdnayhj4v4s7g4rsg9b1jkpdblk91-libcxx-18.1.3/lib,-rpath,/nix/store/1jvrjkymv56pbva6hfsbdavf8h44dn1g-libunwind-18.1.3/lib,-rpath,/nix/store/35pq4hr29c3sl79lgfwgsvd9nwzyp4am-glibc-2.39-5/lib

# Bazel-side configuration for LRE.
build:lre --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build:lre --define=EXECUTOR=remote
build:lre --extra_execution_platforms=@local-remote-execution//generated-cc/config:platform
build:lre --extra_toolchains=@local-remote-execution//generated-cc/config:cc-toolchain
```

The commented paths created in `lre.bazelrc` technically have no effect on what
Bazel's toolchain creates, but for nix to be able to generate the store paths
it needs to fetch the files to your local system. In other words, all paths
printed in `lre.bazelrc` will be available on your local system.

Let's move on to Bazel's configuration.

## 🌱 Bazel-side setup

First, hook the generated `lre.bazelrc` into the main `.bazelrc` with a
`try-import`:

```bash
# In your .bazelrc

# Generated by the LRE flake module.
try-import %workspace%/lre.bazelrc
```

Recall that the `lre.bazelrc` file contains lines like this:

```bash
build:lre --extra_execution_platforms=@local-remote-execution//generated-cc/config:platform
build:lre --extra_toolchains=@local-remote-execution//generated-cc/config:cc-toolchain
```

The `@local-remote-execution` Bazel module contains the Bazel-side configuration
of the LRE toolchain. If you view [`generated-cc/cc/BUILD`](https://github.com/tracemachina/nativelink/tree/main/local-remote-execution/generated-cc/cc/BUILD)
you should find the same `/nix/store/...` paths as in the `lre.bazelrc` file.

Now import the LRE toolchains in your `MODULE.bazel` to make the
`@local-remote-execution` Bazel module available to your build:

```python
# In your MODULE.bazel

bazel_dep(name = "local-remote-execution", version = "0")
git_override(
    module_name = "local-remote-execution",
    remote = "https://github.com/TraceMachina/nativelink",
    strip_prefix = "nativelink",

    # Keep this commit in sync with the one in `flake.nix`.
    commit = "64ed20a40964b8c606c7d65f76af840bcfc837fd",
)
```

Let's use NativeLink's Kubernetes example to verify that the setup worked.

## 🚢 Testing with local K8s

Start the cluster and set up NativeLink in an LRE configuration. For details on
this refer to the [Kubernetes example](https://www.nativelink.com/docs/deployment-examples/kubernetes):

```
/* Detailed source-code truncated for AI context efficiency. */
```

Wait until the cluster is running and the NativeLink deployments are ready. Then
invoke a build against the cluster:

```bash
CACHE=$(kubectl get gtw cache -o=jsonpath='{.status.addresses[0].value}')
SCHEDULER=$(kubectl get gtw scheduler -o=jsonpath='{.status.addresses[0].value}')

# Note: If you don't set `lre.prefix`, you can omit `--config=lre` here as
# LRE will be enabled by default.
bazel build \
    --config=lre \
    --remote_instance_name=main \
    --remote_cache=grpc://$CACHE:50051 \
    --remote_executor=grpc://$SCHEDULER:50052 \
    //local-remote-execution/examples:lre-cc
```

The target builds remotely:

```bash
INFO: Found 1 target...
Target //local-remote-execution/examples:lre-cc up-to-date:
  bazel-bin/local-remote-execution/examples/lre-cc
INFO: Elapsed time: 3.004s, Critical Path: 2.23s
INFO: 11 processes: 9 internal, <<<2 remote>>>.
INFO: Build completed successfully, 11 total actions
```

Now let's disable remote execution and attempt a local rebuild, but keep access
to the remote cache:

```bash
# Clear up any local results.
bazel clean

CACHE=$(kubectl get gtw cache -o=jsonpath='{.status.addresses[0].value}')

# Note: If you don't set `lre.prefix`, you can omit `--config=lre` here as
# LRE will be enabled by default.
bazel build \
    --config=lre \
    --remote_instance_name=main \
    --remote_cache=grpc://$CACHE:50051 \
    //local-remote-execution/examples:lre-cc
```

You'll get a cache hit on the target despite using a local build:

```bash
INFO: Analyzed target //local-remote-execution/examples:lre-cc (91 packages loaded, 1059 targets configured).
INFO: Found 1 target...
Target //local-remote-execution/examples:lre-cc up-to-date:
  bazel-bin/local-remote-execution/examples/lre-cc
INFO: Elapsed time: 0.767s, Critical Path: 0.18s
INFO: 11 processes: <<<2 remote cache hit>>>, 9 internal.
INFO: Build completed successfully, 11 total actions
```

> [!TIP]
> You can also do this the other way around, that is, run a local build and
> upload artifacts to the remote cache and have a remote build reuse those
> artifacts. Or you can only run local builds on different machines that all
> share the same cache.
>
> If you set up all your projects to use the same LRE configuration you'll be
> able to reuse caches for the common dependencies of your projects.

## 🛠️ Rebuilding the toolchains

This is only relevant if you're updating the base toolchains in the `nativelink`
repository itself. If you run `nix flake update` in the `nativelink` repository
you need to update the generated Bazel toolchains as well:

```bash
generate-toolchains
```

## 📐 Architecture

The original C++ and Java toolchain containers are never really instantiated.
Instead, their container environments are used and passed through transforming
functions that take a container schematic as input and generate some other
output.

The first transform is the [`createWorker`](https://github.com/tracemachina/nativelink/tree/main/tools/create-worker.nix) wrapper
which converts an arbitrary OCI image to a NativeLink worker:

```mermaid
flowchart LR
    A(some-toolchain)
        --> |wrap with nativelink| B(nativelink-worker-some-toolchain)
        --> |deploy| K{Kubernetes}
```

In the case of LRE the base image is built with Nix for perfect reproducibility.
However, you could use a more classic toolchain container like an Ubuntu base
image as well:

```mermaid
flowchart LR
    A(lre-cc)
        --> |wrap with nativelink| B(nativelink-worker-lre-cc)
        --> |deploy| K{Kubernetes}
    C(lre-java)
        --> |wrap with nativelink| D(nativelink-worker-lre-java)
        --> |deploy| K{Kubernetes}
```

The second transform is some mechanism that generates toolchain configurations
for your respective RBE client:

```mermaid
flowchart LR
    A(some-toolchain)
        --> |custom toolchain generation logic| B[RBE client configuration]
```

In many classic setups the RBE client configurations are handwritten. In the
case of LRE we're generating Bazel toolchains and platforms using a pipeline of
custom image generators and the `rbe_configs_gen` tool:

```mermaid
flowchart LR
    A(lre-cc)
        --> |wrap with autogen tooling| B(rbe-autogen-lre-cc)
        --> |rbe_configs_gen| C[Bazel C++ toolchain and platform]
```

When you then invoke your RBE client with the configuration set up to use these
toolchains, the NativeLink scheduler matches actions to the worker they require.

In Bazel's case this scheduler endpoint is set via the `--remote_executor` flag.

## 🦜 Custom toolchains

The general approach described works for arbitrary toolchain containers. You
might need to implement your own logic to get from the toolchain container to
some usable RBE configuration files (or write them manually), but this can be
considered an implementation detail specific to your requirements.

TODO(palfrey): Add an example of a custom toolchain extension around lre-cc.

---

## File: nativelink-config/examples/README.md

NativeLink uses a JSON file as the configuration format. This section of the
documentation will breakdown the anatomy of this configuration file with step-by-step instructions on its assembly.

# Assembling the Configuration File

Configuration JSON objects should begin with the `stores` key, followed by `workers`, `schedulers`, `servers`, and `global`.


```json5
{
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {}
}
```

This is the scaffolding for a NativeLink deployment configuration.



  Configuring Store

### Store Name

The value of `stores` is an array where each element defines a store. Each should have a user supplied name. The following example, defines the `AC_MAIN_STORE`.

```json5
{
  "stores": [
    {
      "name": "AC_MAIN_STORE",
    }
  ],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {},
}
```

### Store Type

Once the store has been named and its object exists,
the next key to add is the type of store. The options are `filesystem`, `memory`, `compression`, `dedup`, `fast_slow`, `verify`, and `experimental_cloud_object_store`.

```json5
{
  "stores": [
    {
      "name": "AC_MAIN_STORE",
      "filesystem": {}
    }
  ],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {},
}
```

### Store Options

The contents of the object here must include `content_path`, `temp_path`, and an embedded JSON object, `eviction_policy`, which specifies the value of `max_bytes` for the store.

```json5
{
  "stores": [
    {
      "name": "AC_MAIN_STORE",
      "filesystem": {
        "content_path": "/tmp/nativelink/data/content_path-index",
        "temp_path": "/tmp/nativelink/data/tmp_path-index",
        "eviction_policy": {
          // 500mb.
          "max_bytes": 500000000,
        }
      }
    }
  ],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {},
}
```




  Configuring Workers 

### Worker Array

The value of `workers` includes a top level array that embeds the worker metadata. This array always begins with the `local` object, which is the only item permitted at this time.

```json5
{
  "stores": [],
  "workers": [{
    "local": {}
  }],
  "schedulers": [],
  "servers": [],
  "global": {},
}
```

### Local Worker Object Members

The Local object has five components, `worker_api_endpoint`, `cas_fast_slow_store`, `upload_action_results`,`work_directory`, and `platform_properties`.

```json5
{
  "stores": [],
  "workers": [{
    "local": {
      "worker_api_endpoint": {
        "uri": "grpc://127.0.0.1:50061",
      },
      "cas_fast_slow_store": "WORKER_FAST_SLOW_STORE",
      "upload_action_result": {
        "ac_store": "AC_MAIN_STORE",
      },
      "work_directory": "/tmp/nativelink/work",
      "platform_properties": {
        "cpu_count": {
          "values": ["16"],
        },
        "memory_kb": {
          "values": ["500000"],
        },
        "network_kbps": {
          "values": ["100000"],
        },
        "cpu_arch": {
          "values": ["x86_64"],
        },
        "OSFamily": {
          "values": [""]
        },
        "container-image": {
          "values": [""]
        },
      }
    }
  }],
  "schedulers": [],
  "servers": [],
  "global": {},
}
```




  Configuring Schedulers 

### Scheduler Name
The value of `stores` is an array where each element defines a store. Each should have a user supplied name. The following example, defines the `AC_MAIN_STORE`.

The value of `schedulers` is an array where each element defines a scheduler. Each scheduler must have a user-supplied name. The following example, defines the `MAIN_SCHEDULER`.

```json5
{
  "stores": [],
  "workers": [],
  "schedulers": [
    {
      name: "MAIN_SCHEDULER"
    }
  ],
  "servers": [],
  "global": {},
}
```

### Scheduler Type

Once the scheduler has been named and its object exists,
the next key is the type of scheduler. The options are `simple`, `action_scheduler`, `grpc_scheduler`, `property_modifier_scheduler`, and `worker_scheduler`.

```json5
{
  "stores": [],
  "workers": [],
  "schedulers": [
    {
      name: "MAIN_SCHEDULER",
      simple: {}
    }
  ],
}
```

### Scheduler Options

The contents of the scheduler type object defines the options. For a list of options see the documentation. See the example below.

```json5
{
  "stores": [],
  "workers": [],
  "schedulers": [
    {
      name: "MAIN_SCHEDULER",
      simple: {
        "supported_platform_properties": {
          "cpu_count": "minimum",
          "memory_kb": "minimum",
          "network_kbps": "minimum",
          "disk_read_iops": "minimum",
          "disk_read_bps": "minimum",
          "disk_write_iops": "minimum",
          "disk_write_bps": "minimum",
          "shm_size": "minimum",
          "gpu_count": "minimum",
          "gpu_model": "exact",
          "cpu_vendor": "exact",
          "cpu_arch": "exact",
          "cpu_model": "exact",
          "kernel_version": "exact",
          "OSFamily": "priority",
          "container-image": "priority",
        }
      }
    }
  ],
  "servers": [],
  "global": {},
}
```




  Configuring Servers

### Servers

The `servers` configuration object is an array, with two objects, `public`, and `private_workers_servers`.

```json5
{
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [{
    "name": "public"
  },{
    "name": "private_workers_servers"
  }],
  "global": {},
}
```

### Public Server

The `public` server consists of a `listener` object and a `services` object. The `listener` object is one level of depth and includes an `http` with a `socket address`. The `services` server consists of a `cas`, an `ac`, the `execution`, `capabilities`, and `bytestream`.

```json5
{
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [{
    "name": "public",
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50051"
      }
    },
    "services": {
      "cas": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE"
      }],
      "ac": [{
        "instance_name": "main",
        "ac_store": "AC_MAIN_STORE"
      }],
      "execution": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
        "scheduler": "MAIN_SCHEDULER",
      }],
      "capabilities": [{
        "instance_name": "main",
        "remote_execution": {
          "scheduler": "MAIN_SCHEDULER",
        }
      }],
      "bytestream": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
      }]
    },
  },{
    "name": "private_workers_servers"
  }],
  "global": {},
}
```

### Private Server

> ⚠️ _WARNING_: A private server shouldn't be exposed to the public. ⚠️

The `private` server consists of a `listener` object and a `services` object. The `listener` object is one level and includes an `http` with a `socket address`. The `services` server consists of a `worker_api` object with `scheduler_field` and an `admin` object.

```json5
 {
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [{
    "name": "public",
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50051"
      }
    },
    "services": {
      "cas": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE"
      }],
      "ac": [{
        "instance_name": "main",
        "ac_store": "AC_MAIN_STORE"
      }],
      "execution": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
        "scheduler": "MAIN_SCHEDULER",
      }],
      "capabilities": [{
        "instance_name": "main",
        "remote_execution": {
          "scheduler": "MAIN_SCHEDULER",
        }
      }],
      "bytestream": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
      }]
    },
  },{
    "name": "private_workers_servers",
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50061"
      }
    },
    "services": {
      // Note: This should be served on a different port, because it has
      // a different permission set than the other services.
      // In other words, this service is a backend api. The ones above
      // are a frontend api.
      "worker_api": {
        "scheduler": "MAIN_SCHEDULER",
      },
      "admin": {},
      "health": {
        "timeout_seconds": 2
      },
    }
  }],
  "global": {},
}
```

*`global`* is a single-level object and can be added at the end as the configuration object for file descriptors.

```json5
 "global": {
    "max_open_files": 24576
  }
```




  Complete Example 

Below, you will find a fully tested example that you can also find in [basic_cas.json](basic_cas.json)

```
/* Detailed source-code truncated for AI context efficiency. */
```




  High-Performance tmpfs Configuration

### Using tmpfs for Maximum I/O Performance

NativeLink uses hardlinks to efficiently set up action sandboxes from the CAS filesystem store.
This requires the `work_directory` and the CAS `content_path` to be on the **same filesystem**.

For maximum I/O performance, you can place both on a tmpfs (RAM-based filesystem).
Users have reported **3-4x build time improvements** when using tmpfs.

#### Setup Instructions

1. Create a tmpfs mount point:
```bash
sudo mkdir -p /mnt/tmpfs/nativelink
sudo mount -t tmpfs -o size=50G tmpfs /mnt/tmpfs/nativelink
```

2. To make it persistent across reboots, add to `/etc/fstab`:
```
tmpfs /mnt/tmpfs/nativelink tmpfs size=50G,mode=1777 0 0
```

3. Configure NativeLink to use the tmpfs paths. See [tmpfs-worker.json5](tmpfs-worker.json5) for a complete example.

#### Key Configuration Points

Both paths must be on the same tmpfs mount:
- CAS `content_path`: `/mnt/tmpfs/nativelink/cas`
- Worker `work_directory`: `/mnt/tmpfs/nativelink/work`

#### Trade-offs

| Pros | Cons |
|------|------|
| Maximum I/O performance (RAM speed) | Cache is lost on restart/reboot |
| Hardlinks work correctly (same filesystem) | Limited by available RAM |
| Eliminates disk I/O bottleneck | Not suitable for very large CAS |

#### Sizing Guidance

- Set tmpfs size to ~50% of available RAM
- Set `max_bytes` in eviction_policy to ~80% of tmpfs size
- Monitor memory usage and adjust as needed

---

## File: nativelink-config/README.md

# Configuration examples

NativeLink uses a JSON file as the configuration format.

To view the available fields please refer to the [configuration reference](https://github.com/tracemachina/nativelink/tree/main/nativelink-config/src).

## Examples

The [examples directory](https://github.com/tracemachina/nativelink/tree/master/nativelink-config/examples) contains a few examples of configuration files.

A very basic configuration that's a pure in-memory store is:

```js
{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "memory": {
        "eviction_policy": {
          // 1gb.
          "max_bytes": 1000000000,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "memory": {
        "eviction_policy": {
          // 100mb.
          "max_bytes": 100000000,
        }
      }
    }
  ],
  "servers": [{
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50051",
        "advanced_http": {
          "http2_keep_alive_interval": 10
        }
      }
    },
    "services": {
      "cas": [{
        "instance_name": "main",
        "cas_store": "CAS_MAIN_STORE"
      }],
      "ac": [{
        "instance_name": "main",
        "ac_store": "AC_MAIN_STORE"
      }],
      "capabilities": [{
        "instance_name": "main"
      }],
      "bytestream": [{
        "instance_name": "main",
        "cas_store": "CAS_MAIN_STORE",
      }]
    }
  }]
}
```

### Compression Store

The following configuration will cause the underlying data to be backed by the
filesystem, and when the number of bytes reaches over 100mb for AC objects and
10gb for CAS objects evict them, but apply lz4 compression on the data before
sending it to be stored. This will also automatically decompress the data when
the data is retrieved.

```js
{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "compression": {
        "compression_algorithm": {
          "lz4": {}
        },
        "backend": {
          "filesystem": {
            "content_path": "/tmp/bazel_cache/cas",
            "temp_path": "/tmp/bazel_cache/tmp_data",
            "eviction_policy": {
              // 10gb.
              "max_bytes": 10000000000,
            }
          }
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "filesystem": {
        "content_path": "/tmp/bazel_cache/ac",
        "temp_path": "/tmp/bazel_cache/tmp_data",
        "eviction_policy": {
          // 100mb.
          "max_bytes": 100000000,
        }
      }
    }
  ],
  // Place rest of configuration here ...
}
```

### Dedup Store

In this example we will attempt to de-duplicate our data and compress it before
storing it. This works by applying the [FastCDC](https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf)
window-based rolling checksum algorithm on the data, splitting the data into
smaller pieces then storing each chunk as an individual entry in another store.

This is very useful when large objects are stored and only parts of the
object/file are modified. Examples, are multiple large builds that have debug
information in them. It's very common for large binary objects that contain
debug information to be almost identical when only a subset of modules are
changed. In enterprise level systems this will likely add up to huge
efficiencies, since if just a few bytes are added/removed or changed it will
only transfer the bytes around where the changes occurred.

```
/* Detailed source-code truncated for AI context efficiency. */
```

### S3 Store

Since Amazon's S3 service now has strong consistency, it's very reliable to use
as a back end of a CAS. This pairs well with the `compression` and `dedup`
stores, but in this example we'll store the raw files.

```js
{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "aws",
        // Region the bucket lives in.
        "region": "us-west-1",
        // Name of the bucket to upload to.
        "bucket": "some-bucket-name",
        // Adds an optional prefix to objects before uploaded.
        "key_prefix": "cas/",
        // S3 supports retry capability.
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "aws",
        "region": "us-west-1",
        "bucket": "some-bucket-name",
        "key_prefix": "ac/",
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    }
  ],
  // Place rest of configuration here ...
}
```

### R2 Store

[Cloudflare R2](https://developers.cloudflare.com/r2/) is an S3-compatible
object store with no egress fees, which makes it attractive for read-heavy
CAS workloads. The endpoint is derived from your Cloudflare `account_id`;
credentials are read from env vars via `shellexpand`.

```js
{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "r2",
        "account_id": "your-cloudflare-account-id",
        "bucket": "nativelink-cas",
        "access_key_id": "${R2_ACCESS_KEY_ID}",
        "secret_access_key": "${R2_SECRET_ACCESS_KEY}",
        "key_prefix": "cas/",
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "r2",
        "account_id": "your-cloudflare-account-id",
        "bucket": "nativelink-cas",
        "access_key_id": "${R2_ACCESS_KEY_ID}",
        "secret_access_key": "${R2_SECRET_ACCESS_KEY}",
        "key_prefix": "ac/",
      }
    }
  ],
  // Place rest of configuration here ...
}
```

A complete runnable example with CAS and AC is at
[`nativelink-config/examples/r2_backend.json5`](https://github.com/TraceMachina/nativelink/blob/main/nativelink-config/examples/r2_backend.json5).
If `access_key_id` and `secret_access_key` are omitted, NativeLink falls
back to the standard AWS credential chain (`AWS_*` env vars,
`~/.aws/credentials`, IMDS).

### OCI Store

[Oracle Cloud Infrastructure Object Storage](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
exposes an S3-compatible API, so NativeLink can use it as a CAS/AC backend. The
path-style endpoint is derived from your Object Storage `namespace` and
`region` as
`https://{namespace}.compat.objectstorage.{region}.oci.customer-oci.com`.
Authenticate with a Customer Secret Key (an Access Key/Secret Key pair created
under **User Settings → Customer secret keys** in the OCI console); the secret
cannot be retrieved after generation, so read it from an env var via
`shellexpand`.

```js
{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "oci",
        "namespace": "your-object-storage-namespace",
        "region": "us-phoenix-1",
        "bucket": "nativelink-cas",
        "access_key_id": "${OCI_ACCESS_KEY_ID}",
        "secret_access_key": "${OCI_SECRET_ACCESS_KEY}",
        "key_prefix": "cas/",
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "oci",
        "namespace": "your-object-storage-namespace",
        "region": "us-phoenix-1",
        "bucket": "nativelink-cas",
        "access_key_id": "${OCI_ACCESS_KEY_ID}",
        "secret_access_key": "${OCI_SECRET_ACCESS_KEY}",
        "key_prefix": "ac/",
      }
    }
  ],
  // Place rest of configuration here ...
}
```

A complete runnable example with CAS and AC is at
[`nativelink-config/examples/oci_backend.json5`](https://github.com/TraceMachina/nativelink/blob/main/nativelink-config/examples/oci_backend.json5).
The `region` is used both to build the endpoint host and as the AWS `SigV4`
signing region; if your tooling cannot set an OCI region identifier, OCI also
accepts `us-east-1` to target your tenancy home region. As with the other
S3-compatible stores, omitting `access_key_id` and `secret_access_key` falls
back to the standard AWS credential chain (`AWS_*` env vars).

### Fast Slow Store

This store will first attempt to read from the `fast` store when reading and if
it does exist return it. If it doesn't exist, try to fetch it from the `slow`
store and while streaming it to the client also populate the `fast` store with
the requested object. When transferring (uploading) from client, the data will
be placed into both `fast` and `slow` stores simultaneously.

In this example, we'll hold about 1gb of frequently accessed data in memory and
the rest will be stored in AWS's S3:

```js
{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "fast_slow": {
        "fast": {
          "memory": {
            "eviction_policy": {
              // 1gb.
              "max_bytes": 1000000000,
            }
          }
        },
        "slow": {
          "experimental_cloud_object_store": {
            "provider": "aws",
            "region": "us-west-1",
            "bucket": "some-bucket-name",
            "key_prefix": "cas/",
          }
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "fast_slow": {
        "fast": {
          "memory": {
            "eviction_policy": {
              // 100mb.
              "max_bytes": 100000000,
            }
          }
        },
        "slow": {
          "experimental_cloud_object_store": {
            "provider": "aws",
            "region": "us-west-1",
            "bucket": "some-bucket-name",
            "key_prefix": "ac/",
          }
        }
      }
    }
  ],
  // Place rest of configuration here ...
}
```

### Verify Store

This store is special. It's only job is to verify the content as it's fetched
and uploaded to ensure it meets some criteria or errors. This store should only
be added to the CAS. If `hash_verification_function` is set, it will apply the
hashing algorithm on the data as it's sent/received and at the end if it
doesn't not match the name of the digest it will cancel the upload/download and
return an error. If it's not set, the hashing verification will be disabled.
If `verify_size` is set, a similar item will happen, but count the bytes sent
and check it against the digest instead.

```js
{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "verify": {
        "backend": {
          "memory": {
            "eviction_policy": {
              // 1gb.
              "max_bytes": 1000000000,
            }
          }
        },
        "verify_size": true,
        // sha256 or blake3
        "hash_verification_function": "sha256",
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "memory": {
        "eviction_policy": {
          // 100mb.
          "max_bytes": 100000000,
        }
      }
    }
  ],
  // Place rest of configuration here ...
}
```

---

## File: toolchain-examples/README.md

# Example builds to test remote execution

See: [RBE examples docs](http://nativelink.com/docs/rbe/remote-execution-examples)