nativelink Documentation

NativeLink is a Nix-powered, open source, high-performance build cache and remote execution server, compatible with Bazel, Soong, Pants, Buck2, Reclient,...

1,535 stars

Rust 1 file · ~12,097 tokens #apache2 #bazel #buck2 #build-automation #build-system #chromium #ci #content-addressable-storage

RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy nativelink on any server, homelab, or cloud container platform.

Raw YAML Docker Page → JSON API

1-Click Terminal Command

bash (docker run)

docker run -d --name nativelink -p 4317:4317 --restart always otel/opentelemetry-collector-contrib:0.98.0

Configured Services & Ports

⚙️ otel-collector

image: otel/opentelemetry-collector-contrib:0.98.0

🔌 Ports: 4317:4317, 4318:4318, 9090:9090, 8888:8888, 13133:13133

⚙️ prometheus

image: prom/prometheus:v3.0.0

🔌 Ports: 9091:9090

⚙️ grafana

image: grafana/grafana:12.4.0

🔌 Ports: 3000:3000

⚙️ alertmanager

image: prom/alertmanager:v0.27.0

🔌 Ports: 9093:9093

⚙️ node-exporter

image: prom/node-exporter:v1.7.0

🔌 Ports: 9100:9100

⚙️ jaeger

image: jaegertracing/all-in-one:1.53

🔌 Ports: 16686:16686, 14268:14268

docker-compose.yml

yaml (docker-compose.yml)

version: '3.8'

services:
  # OpenTelemetry Collector
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.98.0
    container_name: otel-collector
    restart: unless-stopped
    command: ["--config=/etc/otel-collector/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector/config.yaml:ro
    ports:
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
      - "9090:9090"   # Prometheus metrics exporter
      - "8888:8888"   # Collector metrics
      - "13133:13133" # Health check
    environment:
      - OTLP_BACKEND_ENDPOINT=${OTLP_BACKEND_ENDPOINT:-otlp-backend:4317}
      - OTLP_BACKEND_TOKEN=${OTLP_BACKEND_TOKEN:-}
    networks:
      - metrics

  # Prometheus with OTLP support
  prometheus:
    image: prom/prometheus:v3.0.0
    container_name: prometheus
    restart: unless-stopped
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--web.enable-otlp-receiver'  # Enable OTLP receiver
      - '--storage.tsdb.retention.time=30d'
    volumes:
      - ./prometheus-config.yaml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus-recording-rules.yml:/etc/prometheus/rules/nativelink.yml:ro
      - prometheus_data:/prometheus
    ports:
      - "9091:9090"  # Prometheus web UI (different port to avoid conflict with collector)
    networks:
      - metrics
    depends_on:
      - otel-collector

  # Grafana for visualization
  grafana:
    image: grafana/grafana:12.4.0
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
    networks:
      - metrics
    depends_on:
      - prometheus

  # Optional: AlertManager for alerts
  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: unless-stopped
    volumes:
      - ./alertmanager-config.yml:/etc/alertmanager/config.yml:ro
      - alertmanager_data:/alertmanager
    ports:
      - "9093:9093"
    command:
      - '--config.file=/etc/alertmanager/config.yml'
      - '--storage.path=/alertmanager'
    networks:
      - metrics

  # Optional: Node exporter for host metrics
  node-exporter:
    image: prom/node-exporter:v1.7.0
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    ports:
      - "9100:9100"
    networks:
      - metrics

  # Optional: Jaeger for trace visualization (if traces are enabled)
  jaeger:
    image: jaegertracing/all-in-one:1.53
    container_name: jaeger
    restart: unless-stopped
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"  # Jaeger UI
      - "14268:14268"  # Jaeger collector HTTP
    networks:
      - metrics

volumes:
  prometheus_data:
  grafana_data:
  alertmanager_data:

networks:
  metrics:
    driver: bridge

# Usage Instructions:
# 1. Start the stack: docker-compose up -d
# 2. Configure NativeLink with these environment variables:
#    export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
#    export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
#    export OTEL_SERVICE_NAME=nativelink
#    export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=dev"
# 3. Access services:
#    - Prometheus: http://localhost:9091
#    - Grafana: http://localhost:3000 (admin/admin)
#    - Jaeger: http://localhost:16686
#    - AlertManager: http://localhost:9093
#    - OTEL Collector metrics: http://localhost:8888/metrics

File: README.md

What's NativeLink?

NativeLink is an efficient, high-performance build cache and remote execution system that accelerates software compilation and testing while reducing infrastructure costs. It optimizes build processes for projects of all sizes by intelligently caching build artifacts and distributing tasks across multiple machines.

NativeLink is trusted in production environments to reduce costs and developer iteration times--handling over billions of requests per month for its customers, including large corporations such as Samsung.

🔑 Key Features

Advanced Build Cache:
- Stores and reuses results of previous build steps for unchanged components
- Significantly reduces build times, especially for incremental changes
Efficient Remote Execution:
- Distributes build and test tasks across a network of machines
- Parallelizes workloads for faster completion
- Utilizes remote resources to offload computational burden from local machines
- Ensures consistency with a uniform, controlled build environment

NativeLink seamlessly integrates with build tools that use the Remote Execution protocol, such as Bazel, Buck2, Goma, and Siso. CMake projects work too via recc. See Build CMake projects with NativeLink. It supports Unix-based operating systems and Windows, ensuring broad compatibility across different development environments.

🚀 Quickstart

To start, you can deploy NativeLink as a Docker image (as shown below). The setups below are production-grade installations. See the contribution docs for instructions on how to build from source with Bazel, Cargo, and Nix.

You can find a few example deployments in the Docs.

📦 Prebuilt images

Fast to spin up, but currently limited to x86_64 systems. See the container
registry
for all image tags and the contribution docs
for how to build the images yourself.

Linux x86_64

bash

curl -O \
    https://raw.githubusercontent.com/TraceMachina/nativelink/v1.4.0/nativelink-config/examples/basic_cas.json5

# See https://github.com/TraceMachina/nativelink/pkgs/container/nativelink
# to find the latest tag
docker run \
    -v $(pwd)/basic_cas.json5:/config \
    -p 50051:50051 \
    ghcr.io/tracemachina/nativelink:v1.4.0 \
    config

Windows x86_64

powershell

# Download the configuration file
Invoke-WebRequest `
    -Uri "https://raw.githubusercontent.com/TraceMachina/nativelink/v1.4.0/nativelink-config/examples/basic_cas.json5" `
    -OutFile "basic_cas.json5"

# Run the Docker container
# Note: Adjust the path if the script is not run from the directory containing basic_cas.json
docker run `
    -v ${PWD}/basic_cas.json5:/config `
    -p 50051:50051 `
    ghcr.io/tracemachina/nativelink:v1.4.0 `
    config

❄️ Raw executable with Nix

Slower, since it's built from source, but more flexible and supports MacOS.
Doesn't support native Windows, but works in WSL2.

Make sure your Nix version is recent and supports flakes. For instance, install
it via the next-gen nix installer.

[!CAUTION]
Executables built for MacOS are dynamically linked against libraries from Nix
and won't work on systems that don't have these libraries present.

[!TIP]
Common setup gotchas for Nix on macOS / Linux:
Active shell environment: If the installer finishes but your shell doesn't recognize nix commands, you need to either restart your terminal session or source the daemon profile manually:
bash
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
Enabling experimental features: If you used the standard Nix installer and get an error saying experimental Nix feature 'nix-command' is disabled, enable them by creating or editing ~/.config/nix/nix.conf:
text
experimental-features = nix-command flakes
Disk Space: Unpacking and building compilers and dependencies requires significant storage. Make sure you have at least 15–20 GB of free space on your system volume before running setup commands.

Linux, MacOS, WSL2

bash

curl -O \
    https://raw.githubusercontent.com/TraceMachina/nativelink/main/nativelink-config/examples/basic_cas.json5

nix run github:TraceMachina/nativelink ./basic_cas.json5

See the contribution docs for further information.

🤝 Contributing

Visit our Contributing guide to learn how to contribute to NativeLink. We welcome contributions from developers of all skill levels and backgrounds!

📊 Stats

📜 License

Licensed under the Functional Source License, Version 1.1, Apache 2.0 Future License.
SPDX identifier: FSL-1.1-Apache-2.0.

After the second anniversary of the date this version was made available, you may use this
software under the Apache License, Version 2.0.

File: deployment-examples/docker-compose/README.md

Multi-Worker Docker Compose Deployment

This guide explains how to run NativeLink with multiple workers using Docker Compose for distributed build execution.

Prerequisites

Architecture: This setup supports x86_64/amd64 and ARM64 architectures, except for Apple Silicon, which is not supported by the test client container
Google Cloud CLI: If using the test client container from gcr.io/bazel-public/bazel, you'll need:
1. Google Cloud SDK installed
2. Authentication via gcloud auth login
3. Docker credential helper configured: gcloud auth configure-docker
Alternatively, you can comment out the test-client service in the docker-compose file and run Bazel tests locally.

Quick Start

Build the Docker container:

docker compose -f
      docker-compose-multi-worker.yml build

Start the multi-worker deployment:

docker compose -f docker-compose-multi-worker.yml up -d

Test with Bazel:

bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052 \
  --remote_default_exec_properties=cpu_count=2

Configuration Requirements

Critical: Shared CAS Storage

All workers MUST share the same CAS storage path. This is the most common configuration error.

Important: When starting workers, you'll see this validation warning:

text

Starting worker 'worker-1'. IMPORTANT: If running multiple workers, all workers
must share the same CAS storage path to avoid 'Object not found' errors.

✅ Correct Configuration:

yaml

services:
  worker-1:
    volumes:
      - cas-data:/data/cas  # Shared volume
  worker-2:
    volumes:
      - cas-data:/data/cas  # Same shared volume
  worker-3:
    volumes:
      - cas-data:/data/cas  # Same shared volume

volumes:
  cas-data:  # Single shared volume for all workers

❌ Incorrect Configuration (causes "Object not found" errors):

yaml

services:
  worker-1:
    volumes:
      - ./data/worker-1/cas:/data/cas  # Isolated path
  worker-2:
    volumes:
      - ./data/worker-2/cas:/data/cas  # Different isolated path

Worker Configuration ([worker-shared-cas.json5](./worker-shared-cas.json5))

json5

{
  stores: [
    {
      name: "SHARED_CAS",
      filesystem: {
        // All workers MUST use the same path
        content_path: "/data/cas/content",
        // Temp can be worker-specific
        temp_path: "/tmp/${WORKER_NAME}/temp",
        eviction_policy: {
          max_bytes: 10000000000,
        }
      }
    }
  ],
  workers: [
    {
      local: {
        name: "${WORKER_NAME}",
        cas_fast_slow_store: "SHARED_CAS",  // All workers use same store
        // Work directory should be worker-specific
        work_directory: "/tmp/${WORKER_NAME}/work",
        platform_properties: {
          cpu_count: { values: ["2"] },
          memory_gb: { values: ["2"] }
        }
      }
    }
  ]
}

Configuration Files

Multi-Worker Setup

docker-compose-multi-worker.yml - Docker Compose file for multi-worker deployment
test-multi-worker-simple.json5 - All-in-one configuration for testing multi-worker setup (validated working)
cas-server-multi-worker.json5 - CAS server configuration for multi-worker deployment
scheduler-multi-worker.json5 - Scheduler configuration for multi-worker deployment
worker-shared-cas.json5 - Worker configuration template with shared CAS storage

Single Worker Setup

docker-compose.yml - Docker Compose file for single worker deployment
local-storage-cas.json5 - Local storage CAS configuration for single worker
scheduler.json5 - Scheduler configuration for single worker deployment
worker.json5 - Worker configuration for single worker deployment

Scaling Workers

Dynamic Scaling

# Scale to 5 workers
docker-compose -f docker-compose-multi-worker.yml up -d --scale worker=5

# Scale down to 2 workers
docker-compose -f docker-compose-multi-worker.yml up -d --scale worker=2

Resource Limits

Each worker is configured with:

CPU: 2 cores (configurable)
Memory: 2GB (configurable)

Adjust in docker-compose-multi-worker.yml:

yaml

deploy:
  resources:
    limits:
      cpus: '4'    # Increase to 4 cores
      memory: 4G   # Increase to 4GB

Monitoring

View Logs

# All workers
docker-compose -f docker-compose-multi-worker.yml logs -f

# Specific worker
docker-compose -f docker-compose-multi-worker.yml logs -f worker-1

# Scheduler
docker-compose -f docker-compose-multi-worker.yml logs -f scheduler

Check Status

# View all containers
docker-compose -f docker-compose-multi-worker.yml ps

# Check CAS storage usage
docker exec -it $(docker-compose -f docker-compose-multi-worker.yml ps -q worker-1) \
  du -sh /data/cas/content

Troubleshooting

"Object not found" Errors

Symptom: Workers fail with enhanced error message:

text

Object 7fd25e01d12373a2d1712e446881c9246a9698da4e7eafecdaeeaaff62195a82-148
not found in either fast or slow store. If using multiple workers, ensure
all workers share the same CAS storage path.

Cause: Workers are using different CAS paths

Solution:

Check volume mounts: docker inspect <worker-container> | grep -A 5 Mounts
Ensure all workers mount the same cas-data volume
Verify content_path is identical in all worker configurations
Use the validated test-multi-worker-simple.json5 as reference

Workers Not Executing Jobs

Symptom: Jobs queue but workers remain idle

Solution:

Check scheduler connectivity: docker logs scheduler | grep -i error
Verify worker registration: docker logs worker-1 | grep "Connected to scheduler"
Check network: docker network ls

Performance Issues

Symptom: Builds slower with multiple workers

Solution:

Check disk I/O: docker stats
Consider using SSD-backed storage
Increase cache size in eviction_policy
Monitor with: docker-compose -f docker-compose-multi-worker.yml logs -f | grep -i slow

Testing Multi-Worker Setup

Local Testing with All-in-One Configuration

Option 1: Using Docker (Recommended)

# Run nativelink using the Docker image you built
docker run --rm -it \
  -v $(pwd)/test-multi-worker-simple.json5:/config.json5 \
  -p 50051:50051 \
  -p 50052:50052 \
  nativelink:latest /config.json5

# In another terminal, test with Bazel
bazel build //:nativelink \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50051 \
  --remote_default_exec_properties=cpu_count=1 \
  --jobs=3

Option 2: Build and Run Locally (Advanced)

# Build nativelink locally first
cargo build --release

# Start multi-worker server (3 workers, shared CAS)
./target/release/nativelink test-multi-worker-simple.json5

# In another terminal, test with Bazel
bazel build //:nativelink \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50051 \
  --remote_default_exec_properties=cpu_count=1 \
  --jobs=3

Basic Test

# Build a simple target
bazel build //src:hello_world \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052

# Check which worker executed
docker-compose -f docker-compose-multi-worker.yml logs | grep "Executing action"

Stress Test

# Build entire project with parallel jobs
bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052 \
  --jobs=50  # High parallelism to distribute across workers

Production Considerations

For production deployments:

Use persistent volumes backed by SSD
Implement health checks in docker-compose.yml
Use external storage (NFS, S3, etc.) for CAS
Monitor worker metrics
Set up log aggregation

See Kubernetes deployment for production-grade configurations.

File: local-remote-execution/README.md

Local Remote Execution

NativeLink's Local Remote Execution is a framework to build, distribute, and
rapidly iterate on custom toolchain setups that are transparent, fully hermetic,
and reproducible across machines of the same system architecture.

Local Remote Execution mirrors toolchains for remote execution in your local
development environment. This lets you reuse build artifacts with virtually
perfect cache hit rate across different repositories, developers, and CI.

[!WARNING]
LRE highly experimental and can be challenging to set up in downstream
projects. Please reach out on the NativeLink
Slack if you have any questions or ideas for improvement.

[!NOTE]
At the moment LRE only works on x86_64-linux.

Pre-Requisites

Nix 2.19.0 or later
A functional local container setup

❄️ Flake-side setup

Your project must have a Nix flake and use flake-parts.

Add the nativelink repository to your flake inputs and make the nixpkgs from
nativelink the global input for your project:

nix

inputs = {
  nixpkgs = {
    url = "github:nixos/nixpkgs";

    # IMPORTANT: This needs to follow the `nixpkgs` from nativelink so that
    # the local LRE toolchains are in sync with the remote toolchains.
    follows = "nativelink/nixpkgs";
  };
  nativelink = {
    # This repository provides the autogenerated LRE toolchains which are
    # dependent on the nixpkgs version in the nativelink repository. To keep the
    # local LRE toolchains aligned with remote LRE, we need to use the nixpkgs
    # used by nativelink as the the "global" nixpkgs. We do this by setting
    # `nixpkgs.follows = "nativelink/nixpkgs"` above.

    # Keep this commit in sync with the LRE commit that you import via Bazel.
    url = "github:TraceMachina/nativelink/64ed20a40964b8c606c7d65f76af840bcfc837fd";
  };
  # ...
};

Then, in your flake-parts imports, add the nativelink flake module:

nix

imports = [
  inputs.nativelink.flakeModule
];

Finally, add the lre.bazelrc generator in your devShell's shellHook:

nix

devShells.default = pkgs.mkShell {
  # ...
  shellHook = ''
    # Generate lre.bazelrc which configures LRE toolchains when running
    # in the nix environment.
    ${config.lre.installationScript}
  '';
}

Now, when you enter the nix flake (automated via direnv or manually via
nix develop) it will generate a file called lre.bazelrc which contains
information about the LRE toolchain and configures Bazel. With the default
configuration it looks like this:

bash

# These flags are dynamically generated by the LRE flake module.
#
# Add `try-import %workspace%/lre.bazelrc` to your .bazelrc to
# include these flags when running Bazel in a nix environment.

# These are the paths used by your local LRE config. If you get cache misses
# between local and remote execution, double-check these values against the
# toolchain configs in the `@local-remote-execution` repository at the
# commit that you imported in your `MODULE.bazel`.
#
# WARNING: No environment set. LRE will not work locally.

# Bazel-side configuration for LRE.
build --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build --define=EXECUTOR=remote
build --extra_execution_platforms=@local-remote-execution//generated-cc/config:platform
build --extra_toolchains=@local-remote-execution//generated-cc/config:cc-toolchain

In the snippet above you can see a warning that no local toolchain is
configured. LRE needs to know the remote toolchain configuration to make it
available locally. The lre settings take an Env input and
an optional prefix input to configure the generated lre.bazelrc:

nix

# This is a top-level field, next to `packages` and `apps`, and `devShells`
lre = {
  # In this example we import the lre-cc environment from nativelink and make it
  # available locally.
  inherit (lre-cc.meta) Env;

  # This causes the `build` commands in the `lre.bazelrc` to be prefixed with
  # `lre` so that LRE is disabled by default and may be enabled by passing
  # `--config=lre` to a Bazel build.
  prefix = "lre";
};

Exit and re-enter the flake. The lre.bazelrc should look now like this:

bash

# These flags are dynamically generated by the LRE flake module.
#
# Add `try-import %workspace%/lre.bazelrc` to your .bazelrc to
# include these flags when running Bazel in a nix environment.

# These are the paths used by your local LRE config. If you get cache misses
# between local and remote execution, double-check these values against the
# toolchain configs in the `@local-remote-execution` repository at the
# commit that you imported in your `MODULE.bazel`.
#
# PATH=/nix/store/lr2xhk791gy0lfwk4g5954kd8yfqsbsl-binutils-wrapper-2.41/bin:/nix/store/29xdc6ax5g41njf20h09gmf52qm1zws6-customClang/bin:/nix/store/kwlg5k0zmnkkj3p2kzi3b49sj73887j4-stdenv-linux/bin:/nix/store/asqa3kfq3maclk7cqqhrjvp7vriw6ahy-coreutils-9.5/bin:/nix/store/vf8cjkgwmgd4sb3vkxh6x9iar71s1w1c-findutils-4.9.0/bin:/nix/store/zs4ajc6bp81c92fj2l5r2d68ra0hmfbb-gnutar-1.35/bin
# CC=/nix/store/29xdc6ax5g41njf20h09gmf52qm1zws6-customClang/bin/customClang
# BAZEL_LINKOPTS=-L/nix/store/z56vdnayhj4v4s7g4rsg9b1jkpdblk91-libcxx-18.1.3/lib:-L/nix/store/1jvrjkymv56pbva6hfsbdavf8h44dn1g-libunwind-18.1.3/lib:-lc++:-Wl,-rpath,/nix/store/z56vdnayhj4v4s7g4rsg9b1jkpdblk91-libcxx-18.1.3/lib,-rpath,/nix/store/1jvrjkymv56pbva6hfsbdavf8h44dn1g-libunwind-18.1.3/lib,-rpath,/nix/store/35pq4hr29c3sl79lgfwgsvd9nwzyp4am-glibc-2.39-5/lib

# Bazel-side configuration for LRE.
build:lre --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build:lre --define=EXECUTOR=remote
build:lre --extra_execution_platforms=@local-remote-execution//generated-cc/config:platform
build:lre --extra_toolchains=@local-remote-execution//generated-cc/config:cc-toolchain

The commented paths created in lre.bazelrc technically have no effect on what
Bazel's toolchain creates, but for nix to be able to generate the store paths
it needs to fetch the files to your local system. In other words, all paths
printed in lre.bazelrc will be available on your local system.

Let's move on to Bazel's configuration.

🌱 Bazel-side setup

First, hook the generated lre.bazelrc into the main .bazelrc with a
try-import:

bash

# In your .bazelrc

# Generated by the LRE flake module.
try-import %workspace%/lre.bazelrc

Recall that the lre.bazelrc file contains lines like this:

bash

build:lre --extra_execution_platforms=@local-remote-execution//generated-cc/config:platform
build:lre --extra_toolchains=@local-remote-execution//generated-cc/config:cc-toolchain

The @local-remote-execution Bazel module contains the Bazel-side configuration
of the LRE toolchain. If you view generated-cc/cc/BUILD
you should find the same /nix/store/... paths as in the lre.bazelrc file.

Now import the LRE toolchains in your MODULE.bazel to make the
@local-remote-execution Bazel module available to your build:

python

# In your MODULE.bazel

bazel_dep(name = "local-remote-execution", version = "0")
git_override(
    module_name = "local-remote-execution",
    remote = "https://github.com/TraceMachina/nativelink",
    strip_prefix = "nativelink",

    # Keep this commit in sync with the one in `flake.nix`.
    commit = "64ed20a40964b8c606c7d65f76af840bcfc837fd",
)

Let's use NativeLink's Kubernetes example to verify that the setup worked.

🚢 Testing with local K8s

Start the cluster and set up NativeLink in an LRE configuration. For details on
this refer to the Kubernetes example:

text

/* Detailed source-code truncated for AI context efficiency. */

Wait until the cluster is running and the NativeLink deployments are ready. Then
invoke a build against the cluster:

bash

CACHE=$(kubectl get gtw cache -o=jsonpath='{.status.addresses[0].value}')
SCHEDULER=$(kubectl get gtw scheduler -o=jsonpath='{.status.addresses[0].value}')

# Note: If you don't set `lre.prefix`, you can omit `--config=lre` here as
# LRE will be enabled by default.
bazel build \
    --config=lre \
    --remote_instance_name=main \
    --remote_cache=grpc://$CACHE:50051 \
    --remote_executor=grpc://$SCHEDULER:50052 \
    //local-remote-execution/examples:lre-cc

The target builds remotely:

bash

INFO: Found 1 target...
Target //local-remote-execution/examples:lre-cc up-to-date:
  bazel-bin/local-remote-execution/examples/lre-cc
INFO: Elapsed time: 3.004s, Critical Path: 2.23s
INFO: 11 processes: 9 internal, <<<2 remote>>>.
INFO: Build completed successfully, 11 total actions

Now let's disable remote execution and attempt a local rebuild, but keep access
to the remote cache:

bash

# Clear up any local results.
bazel clean

CACHE=$(kubectl get gtw cache -o=jsonpath='{.status.addresses[0].value}')

# Note: If you don't set `lre.prefix`, you can omit `--config=lre` here as
# LRE will be enabled by default.
bazel build \
    --config=lre \
    --remote_instance_name=main \
    --remote_cache=grpc://$CACHE:50051 \
    //local-remote-execution/examples:lre-cc

You'll get a cache hit on the target despite using a local build:

bash

INFO: Analyzed target //local-remote-execution/examples:lre-cc (91 packages loaded, 1059 targets configured).
INFO: Found 1 target...
Target //local-remote-execution/examples:lre-cc up-to-date:
  bazel-bin/local-remote-execution/examples/lre-cc
INFO: Elapsed time: 0.767s, Critical Path: 0.18s
INFO: 11 processes: <<<2 remote cache hit>>>, 9 internal.
INFO: Build completed successfully, 11 total actions

[!TIP]
You can also do this the other way around, that is, run a local build and
upload artifacts to the remote cache and have a remote build reuse those
artifacts. Or you can only run local builds on different machines that all
share the same cache.

If you set up all your projects to use the same LRE configuration you'll be
able to reuse caches for the common dependencies of your projects.

🛠️ Rebuilding the toolchains

This is only relevant if you're updating the base toolchains in the nativelink
repository itself. If you run nix flake update in the nativelink repository
you need to update the generated Bazel toolchains as well:

bash

generate-toolchains

📐 Architecture

The original C++ and Java toolchain containers are never really instantiated.
Instead, their container environments are used and passed through transforming
functions that take a container schematic as input and generate some other
output.

The first transform is the createWorker wrapper
which converts an arbitrary OCI image to a NativeLink worker:

mermaid

flowchart LR
    A(some-toolchain)
        --> |wrap with nativelink| B(nativelink-worker-some-toolchain)
        --> |deploy| K{Kubernetes}

In the case of LRE the base image is built with Nix for perfect reproducibility.
However, you could use a more classic toolchain container like an Ubuntu base
image as well:

mermaid

flowchart LR
    A(lre-cc)
        --> |wrap with nativelink| B(nativelink-worker-lre-cc)
        --> |deploy| K{Kubernetes}
    C(lre-java)
        --> |wrap with nativelink| D(nativelink-worker-lre-java)
        --> |deploy| K{Kubernetes}

The second transform is some mechanism that generates toolchain configurations
for your respective RBE client:

mermaid

flowchart LR
    A(some-toolchain)
        --> |custom toolchain generation logic| B[RBE client configuration]

In many classic setups the RBE client configurations are handwritten. In the
case of LRE we're generating Bazel toolchains and platforms using a pipeline of
custom image generators and the rbe_configs_gen tool:

mermaid

flowchart LR
    A(lre-cc)
        --> |wrap with autogen tooling| B(rbe-autogen-lre-cc)
        --> |rbe_configs_gen| C[Bazel C++ toolchain and platform]

When you then invoke your RBE client with the configuration set up to use these
toolchains, the NativeLink scheduler matches actions to the worker they require.

In Bazel's case this scheduler endpoint is set via the --remote_executor flag.

🦜 Custom toolchains

The general approach described works for arbitrary toolchain containers. You
might need to implement your own logic to get from the toolchain container to
some usable RBE configuration files (or write them manually), but this can be
considered an implementation detail specific to your requirements.

TODO(palfrey): Add an example of a custom toolchain extension around lre-cc.

File: nativelink-config/examples/README.md

NativeLink uses a JSON file as the configuration format. This section of the
documentation will breakdown the anatomy of this configuration file with step-by-step instructions on its assembly.

Assembling the Configuration File

Configuration JSON objects should begin with the stores key, followed by workers, schedulers, servers, and global.

json5

{
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {}
}

This is the scaffolding for a NativeLink deployment configuration.

Configuring Store

Store Name

The value of stores is an array where each element defines a store. Each should have a user supplied name. The following example, defines the AC_MAIN_STORE.

json5

{
  "stores": [
    {
      "name": "AC_MAIN_STORE",
    }
  ],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {},
}

Store Type

Once the store has been named and its object exists,
the next key to add is the type of store. The options are filesystem, memory, compression, dedup, fast_slow, verify, and experimental_cloud_object_store.

json5

{
  "stores": [
    {
      "name": "AC_MAIN_STORE",
      "filesystem": {}
    }
  ],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {},
}

Store Options

The contents of the object here must include content_path, temp_path, and an embedded JSON object, eviction_policy, which specifies the value of max_bytes for the store.

json5

{
  "stores": [
    {
      "name": "AC_MAIN_STORE",
      "filesystem": {
        "content_path": "/tmp/nativelink/data/content_path-index",
        "temp_path": "/tmp/nativelink/data/tmp_path-index",
        "eviction_policy": {
          // 500mb.
          "max_bytes": 500000000,
        }
      }
    }
  ],
  "workers": [],
  "schedulers": [],
  "servers": [],
  "global": {},
}

Configuring Workers

Worker Array

The value of workers includes a top level array that embeds the worker metadata. This array always begins with the local object, which is the only item permitted at this time.

json5

{
  "stores": [],
  "workers": [{
    "local": {}
  }],
  "schedulers": [],
  "servers": [],
  "global": {},
}

Local Worker Object Members

The Local object has five components, worker_api_endpoint, cas_fast_slow_store, upload_action_results,work_directory, and platform_properties.

json5

{
  "stores": [],
  "workers": [{
    "local": {
      "worker_api_endpoint": {
        "uri": "grpc://127.0.0.1:50061",
      },
      "cas_fast_slow_store": "WORKER_FAST_SLOW_STORE",
      "upload_action_result": {
        "ac_store": "AC_MAIN_STORE",
      },
      "work_directory": "/tmp/nativelink/work",
      "platform_properties": {
        "cpu_count": {
          "values": ["16"],
        },
        "memory_kb": {
          "values": ["500000"],
        },
        "network_kbps": {
          "values": ["100000"],
        },
        "cpu_arch": {
          "values": ["x86_64"],
        },
        "OSFamily": {
          "values": [""]
        },
        "container-image": {
          "values": [""]
        },
      }
    }
  }],
  "schedulers": [],
  "servers": [],
  "global": {},
}

Configuring Schedulers

Scheduler Name

The value of stores is an array where each element defines a store. Each should have a user supplied name. The following example, defines the AC_MAIN_STORE.

The value of schedulers is an array where each element defines a scheduler. Each scheduler must have a user-supplied name. The following example, defines the MAIN_SCHEDULER.

json5

{
  "stores": [],
  "workers": [],
  "schedulers": [
    {
      name: "MAIN_SCHEDULER"
    }
  ],
  "servers": [],
  "global": {},
}

Scheduler Type

Once the scheduler has been named and its object exists,
the next key is the type of scheduler. The options are simple, action_scheduler, grpc_scheduler, property_modifier_scheduler, and worker_scheduler.

json5

{
  "stores": [],
  "workers": [],
  "schedulers": [
    {
      name: "MAIN_SCHEDULER",
      simple: {}
    }
  ],
}

Scheduler Options

The contents of the scheduler type object defines the options. For a list of options see the documentation. See the example below.

json5

{
  "stores": [],
  "workers": [],
  "schedulers": [
    {
      name: "MAIN_SCHEDULER",
      simple: {
        "supported_platform_properties": {
          "cpu_count": "minimum",
          "memory_kb": "minimum",
          "network_kbps": "minimum",
          "disk_read_iops": "minimum",
          "disk_read_bps": "minimum",
          "disk_write_iops": "minimum",
          "disk_write_bps": "minimum",
          "shm_size": "minimum",
          "gpu_count": "minimum",
          "gpu_model": "exact",
          "cpu_vendor": "exact",
          "cpu_arch": "exact",
          "cpu_model": "exact",
          "kernel_version": "exact",
          "OSFamily": "priority",
          "container-image": "priority",
        }
      }
    }
  ],
  "servers": [],
  "global": {},
}

Configuring Servers

Servers

The servers configuration object is an array, with two objects, public, and private_workers_servers.

json5

{
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [{
    "name": "public"
  },{
    "name": "private_workers_servers"
  }],
  "global": {},
}

Public Server

The public server consists of a listener object and a services object. The listener object is one level of depth and includes an http with a socket address. The services server consists of a cas, an ac, the execution, capabilities, and bytestream.

json5

{
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [{
    "name": "public",
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50051"
      }
    },
    "services": {
      "cas": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE"
      }],
      "ac": [{
        "instance_name": "main",
        "ac_store": "AC_MAIN_STORE"
      }],
      "execution": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
        "scheduler": "MAIN_SCHEDULER",
      }],
      "capabilities": [{
        "instance_name": "main",
        "remote_execution": {
          "scheduler": "MAIN_SCHEDULER",
        }
      }],
      "bytestream": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
      }]
    },
  },{
    "name": "private_workers_servers"
  }],
  "global": {},
}

Private Server

⚠️ WARNING: A private server shouldn't be exposed to the public. ⚠️

The private server consists of a listener object and a services object. The listener object is one level and includes an http with a socket address. The services server consists of a worker_api object with scheduler_field and an admin object.

json5

{
  "stores": [],
  "workers": [],
  "schedulers": [],
  "servers": [{
    "name": "public",
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50051"
      }
    },
    "services": {
      "cas": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE"
      }],
      "ac": [{
        "instance_name": "main",
        "ac_store": "AC_MAIN_STORE"
      }],
      "execution": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
        "scheduler": "MAIN_SCHEDULER",
      }],
      "capabilities": [{
        "instance_name": "main",
        "remote_execution": {
          "scheduler": "MAIN_SCHEDULER",
        }
      }],
      "bytestream": [{
        "instance_name": "main",
        "cas_store": "WORKER_FAST_SLOW_STORE",
      }]
    },
  },{
    "name": "private_workers_servers",
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50061"
      }
    },
    "services": {
      // Note: This should be served on a different port, because it has
      // a different permission set than the other services.
      // In other words, this service is a backend api. The ones above
      // are a frontend api.
      "worker_api": {
        "scheduler": "MAIN_SCHEDULER",
      },
      "admin": {},
      "health": {
        "timeout_seconds": 2
      },
    }
  }],
  "global": {},
}

global is a single-level object and can be added at the end as the configuration object for file descriptors.

json5

"global": {
    "max_open_files": 24576
  }

Complete Example

Below, you will find a fully tested example that you can also find in basic_cas.json

text

/* Detailed source-code truncated for AI context efficiency. */

High-Performance tmpfs Configuration

Using tmpfs for Maximum I/O Performance

NativeLink uses hardlinks to efficiently set up action sandboxes from the CAS filesystem store.
This requires the work_directory and the CAS content_path to be on the same filesystem.

For maximum I/O performance, you can place both on a tmpfs (RAM-based filesystem).
Users have reported 3-4x build time improvements when using tmpfs.

Setup Instructions

Create a tmpfs mount point:

bash

sudo mkdir -p /mnt/tmpfs/nativelink
sudo mount -t tmpfs -o size=50G tmpfs /mnt/tmpfs/nativelink

To make it persistent across reboots, add to /etc/fstab:

text

tmpfs /mnt/tmpfs/nativelink tmpfs size=50G,mode=1777 0 0

Configure NativeLink to use the tmpfs paths. See tmpfs-worker.json5 for a complete example.

Key Configuration Points

Both paths must be on the same tmpfs mount:

CAS content_path: /mnt/tmpfs/nativelink/cas
Worker work_directory: /mnt/tmpfs/nativelink/work

Trade-offs

Pros	Cons
Maximum I/O performance (RAM speed)	Cache is lost on restart/reboot
Hardlinks work correctly (same filesystem)	Limited by available RAM
Eliminates disk I/O bottleneck	Not suitable for very large CAS

Sizing Guidance

Set tmpfs size to ~50% of available RAM
Set max_bytes in eviction_policy to ~80% of tmpfs size
Monitor memory usage and adjust as needed

File: nativelink-config/README.md

Configuration examples

NativeLink uses a JSON file as the configuration format.

To view the available fields please refer to the configuration reference.

Examples

The examples directory contains a few examples of configuration files.

A very basic configuration that's a pure in-memory store is:

{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "memory": {
        "eviction_policy": {
          // 1gb.
          "max_bytes": 1000000000,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "memory": {
        "eviction_policy": {
          // 100mb.
          "max_bytes": 100000000,
        }
      }
    }
  ],
  "servers": [{
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50051",
        "advanced_http": {
          "http2_keep_alive_interval": 10
        }
      }
    },
    "services": {
      "cas": [{
        "instance_name": "main",
        "cas_store": "CAS_MAIN_STORE"
      }],
      "ac": [{
        "instance_name": "main",
        "ac_store": "AC_MAIN_STORE"
      }],
      "capabilities": [{
        "instance_name": "main"
      }],
      "bytestream": [{
        "instance_name": "main",
        "cas_store": "CAS_MAIN_STORE",
      }]
    }
  }]
}

Compression Store

The following configuration will cause the underlying data to be backed by the
filesystem, and when the number of bytes reaches over 100mb for AC objects and
10gb for CAS objects evict them, but apply lz4 compression on the data before
sending it to be stored. This will also automatically decompress the data when
the data is retrieved.

{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "compression": {
        "compression_algorithm": {
          "lz4": {}
        },
        "backend": {
          "filesystem": {
            "content_path": "/tmp/bazel_cache/cas",
            "temp_path": "/tmp/bazel_cache/tmp_data",
            "eviction_policy": {
              // 10gb.
              "max_bytes": 10000000000,
            }
          }
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "filesystem": {
        "content_path": "/tmp/bazel_cache/ac",
        "temp_path": "/tmp/bazel_cache/tmp_data",
        "eviction_policy": {
          // 100mb.
          "max_bytes": 100000000,
        }
      }
    }
  ],
  // Place rest of configuration here ...
}

Dedup Store

In this example we will attempt to de-duplicate our data and compress it before
storing it. This works by applying the FastCDC
window-based rolling checksum algorithm on the data, splitting the data into
smaller pieces then storing each chunk as an individual entry in another store.

This is very useful when large objects are stored and only parts of the
object/file are modified. Examples, are multiple large builds that have debug
information in them. It's very common for large binary objects that contain
debug information to be almost identical when only a subset of modules are
changed. In enterprise level systems this will likely add up to huge
efficiencies, since if just a few bytes are added/removed or changed it will
only transfer the bytes around where the changes occurred.

text

/* Detailed source-code truncated for AI context efficiency. */

S3 Store

Since Amazon's S3 service now has strong consistency, it's very reliable to use
as a back end of a CAS. This pairs well with the compression and dedup
stores, but in this example we'll store the raw files.

{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "aws",
        // Region the bucket lives in.
        "region": "us-west-1",
        // Name of the bucket to upload to.
        "bucket": "some-bucket-name",
        // Adds an optional prefix to objects before uploaded.
        "key_prefix": "cas/",
        // S3 supports retry capability.
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "aws",
        "region": "us-west-1",
        "bucket": "some-bucket-name",
        "key_prefix": "ac/",
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    }
  ],
  // Place rest of configuration here ...
}

R2 Store

Cloudflare R2 is an S3-compatible
object store with no egress fees, which makes it attractive for read-heavy
CAS workloads. The endpoint is derived from your Cloudflare account_id;
credentials are read from env vars via shellexpand.

{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "r2",
        "account_id": "your-cloudflare-account-id",
        "bucket": "nativelink-cas",
        "access_key_id": "${R2_ACCESS_KEY_ID}",
        "secret_access_key": "${R2_SECRET_ACCESS_KEY}",
        "key_prefix": "cas/",
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "r2",
        "account_id": "your-cloudflare-account-id",
        "bucket": "nativelink-cas",
        "access_key_id": "${R2_ACCESS_KEY_ID}",
        "secret_access_key": "${R2_SECRET_ACCESS_KEY}",
        "key_prefix": "ac/",
      }
    }
  ],
  // Place rest of configuration here ...
}

A complete runnable example with CAS and AC is at
nativelink-config/examples/r2_backend.json5.
If access_key_id and secret_access_key are omitted, NativeLink falls
back to the standard AWS credential chain (AWS_* env vars,
~/.aws/credentials, IMDS).

OCI Store

Oracle Cloud Infrastructure Object Storage
exposes an S3-compatible API, so NativeLink can use it as a CAS/AC backend. The
path-style endpoint is derived from your Object Storage namespace and
region as
https://{namespace}.compat.objectstorage.{region}.oci.customer-oci.com.
Authenticate with a Customer Secret Key (an Access Key/Secret Key pair created
under User Settings → Customer secret keys in the OCI console); the secret
cannot be retrieved after generation, so read it from an env var via
shellexpand.

{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "oci",
        "namespace": "your-object-storage-namespace",
        "region": "us-phoenix-1",
        "bucket": "nativelink-cas",
        "access_key_id": "${OCI_ACCESS_KEY_ID}",
        "secret_access_key": "${OCI_SECRET_ACCESS_KEY}",
        "key_prefix": "cas/",
        "retry": {
          "max_retries": 6,
          "delay": 0.3,
          "jitter": 0.5,
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "experimental_cloud_object_store": {
        "provider": "oci",
        "namespace": "your-object-storage-namespace",
        "region": "us-phoenix-1",
        "bucket": "nativelink-cas",
        "access_key_id": "${OCI_ACCESS_KEY_ID}",
        "secret_access_key": "${OCI_SECRET_ACCESS_KEY}",
        "key_prefix": "ac/",
      }
    }
  ],
  // Place rest of configuration here ...
}

A complete runnable example with CAS and AC is at
nativelink-config/examples/oci_backend.json5.
The region is used both to build the endpoint host and as the AWS SigV4
signing region; if your tooling cannot set an OCI region identifier, OCI also
accepts us-east-1 to target your tenancy home region. As with the other
S3-compatible stores, omitting access_key_id and secret_access_key falls
back to the standard AWS credential chain (AWS_* env vars).

Fast Slow Store

This store will first attempt to read from the fast store when reading and if
it does exist return it. If it doesn't exist, try to fetch it from the slow
store and while streaming it to the client also populate the fast store with
the requested object. When transferring (uploading) from client, the data will
be placed into both fast and slow stores simultaneously.

In this example, we'll hold about 1gb of frequently accessed data in memory and
the rest will be stored in AWS's S3:

{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "fast_slow": {
        "fast": {
          "memory": {
            "eviction_policy": {
              // 1gb.
              "max_bytes": 1000000000,
            }
          }
        },
        "slow": {
          "experimental_cloud_object_store": {
            "provider": "aws",
            "region": "us-west-1",
            "bucket": "some-bucket-name",
            "key_prefix": "cas/",
          }
        }
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "fast_slow": {
        "fast": {
          "memory": {
            "eviction_policy": {
              // 100mb.
              "max_bytes": 100000000,
            }
          }
        },
        "slow": {
          "experimental_cloud_object_store": {
            "provider": "aws",
            "region": "us-west-1",
            "bucket": "some-bucket-name",
            "key_prefix": "ac/",
          }
        }
      }
    }
  ],
  // Place rest of configuration here ...
}

Verify Store

This store is special. It's only job is to verify the content as it's fetched
and uploaded to ensure it meets some criteria or errors. This store should only
be added to the CAS. If hash_verification_function is set, it will apply the
hashing algorithm on the data as it's sent/received and at the end if it
doesn't not match the name of the digest it will cancel the upload/download and
return an error. If it's not set, the hashing verification will be disabled.
If verify_size is set, a similar item will happen, but count the bytes sent
and check it against the digest instead.

{
  "stores": [
    {
      "name": "CAS_MAIN_STORE",
      "verify": {
        "backend": {
          "memory": {
            "eviction_policy": {
              // 1gb.
              "max_bytes": 1000000000,
            }
          }
        },
        "verify_size": true,
        // sha256 or blake3
        "hash_verification_function": "sha256",
      }
    },
    {
      "name": "AC_MAIN_STORE",
      "memory": {
        "eviction_policy": {
          // 100mb.
          "max_bytes": 100000000,
        }
      }
    }
  ],
  // Place rest of configuration here ...
}

File: toolchain-examples/README.md

Example builds to test remote execution

See: RBE examples docs

Docker & Self-Hosted Deployment Verified OCI

File: README.md Copy Chapter

What's NativeLink? Copy Chapter

🔑 Key Features Copy Chapter

🚀 Quickstart Copy Chapter

📦 Prebuilt images Copy Section

❄️ Raw executable with Nix Copy Section

🤝 Contributing Copy Chapter

📊 Stats Copy Chapter

📜 License Copy Chapter

File: deployment-examples/docker-compose/README.md Copy Chapter

Multi-Worker Docker Compose Deployment

Prerequisites Copy Chapter

Quick Start Copy Chapter

Configuration Requirements Copy Chapter

Critical: Shared CAS Storage Copy Section

Worker Configuration ([worker-shared-cas.json5](./worker-shared-cas.json5)) Copy Section

Configuration Files Copy Chapter

Multi-Worker Setup Copy Section

Single Worker Setup Copy Section

Scaling Workers Copy Chapter

Dynamic Scaling Copy Section

Resource Limits Copy Section

Monitoring Copy Chapter

View Logs Copy Section

Check Status Copy Section

Troubleshooting Copy Chapter

"Object not found" Errors Copy Section

Workers Not Executing Jobs Copy Section

Performance Issues Copy Section

Testing Multi-Worker Setup Copy Chapter

Local Testing with All-in-One Configuration Copy Section

Option 1: Using Docker (Recommended)

Option 2: Build and Run Locally (Advanced)

Basic Test Copy Section

Stress Test Copy Section

Production Considerations Copy Chapter

File: local-remote-execution/README.md Copy Chapter

Local Remote Execution

Pre-Requisites Copy Chapter

❄️ Flake-side setup Copy Chapter

🌱 Bazel-side setup Copy Chapter

🚢 Testing with local K8s Copy Chapter

🛠️ Rebuilding the toolchains Copy Chapter

📐 Architecture Copy Chapter

🦜 Custom toolchains Copy Chapter

File: nativelink-config/examples/README.md Copy Chapter

Assembling the Configuration File

Store Name Copy Section

Store Type Copy Section

Store Options Copy Section

Worker Array Copy Section

Local Worker Object Members Copy Section

Scheduler Name Copy Section

Scheduler Type Copy Section

Scheduler Options Copy Section

Servers Copy Section

Public Server Copy Section

Private Server Copy Section

Using tmpfs for Maximum I/O Performance Copy Section

Setup Instructions

Key Configuration Points

Trade-offs

Sizing Guidance

File: nativelink-config/README.md Copy Chapter

Configuration examples

Examples Copy Chapter

Compression Store Copy Section

Dedup Store Copy Section

S3 Store Copy Section

R2 Store Copy Section

OCI Store Copy Section

Fast Slow Store Copy Section

Verify Store Copy Section

File: toolchain-examples/README.md Copy Chapter

Example builds to test remote execution

Explore Related Technologies & Topics

File: README.md

What's NativeLink?

🔑 Key Features

🚀 Quickstart

📦 Prebuilt images

❄️ Raw executable with Nix

🤝 Contributing

📊 Stats

📜 License

File: deployment-examples/docker-compose/README.md

Prerequisites

Quick Start

Configuration Requirements

Critical: Shared CAS Storage

Worker Configuration ([worker-shared-cas.json5](./worker-shared-cas.json5))

Configuration Files

Multi-Worker Setup

Single Worker Setup

Scaling Workers

Dynamic Scaling

Resource Limits

Monitoring

View Logs

Check Status

Troubleshooting

"Object not found" Errors

Workers Not Executing Jobs

Performance Issues

Testing Multi-Worker Setup

Local Testing with All-in-One Configuration

Basic Test

Stress Test

Production Considerations

File: local-remote-execution/README.md

Pre-Requisites

❄️ Flake-side setup

🌱 Bazel-side setup

🚢 Testing with local K8s

🛠️ Rebuilding the toolchains

📐 Architecture

🦜 Custom toolchains

File: nativelink-config/examples/README.md

Store Name

Store Type

Store Options

Worker Array

Local Worker Object Members

Scheduler Name

Scheduler Type

Scheduler Options

Servers

Public Server

Private Server

Using tmpfs for Maximum I/O Performance

File: nativelink-config/README.md

Examples

Compression Store

Dedup Store

S3 Store

R2 Store

OCI Store

Fast Slow Store

Verify Store

File: toolchain-examples/README.md