snarkOS Documentation

Table of Contents Copy Chapter

• 1. Overview
• 2. Build Guide
  • 2.1 Requirements
  • 2.2 Installation
• 3. Run an Aleo Node
  • 3.1 Run an Aleo Client
  • 3.2 Run an Aleo Validator
  • 3.3 Run an Aleo Prover
• 4. FAQs
• 5. Command Line Interface
• 6. Development Guide
  • 6.1 Quick Start
  • 6.2 Operations
  • 6.3 Local Devnet
  • 6.4 Feature Flags
  • 6.5 Local Backups
• 7. Contributors
• 8. License

1. Overview Copy Chapter

snarkOS is a decentralized operating system for zero-knowledge applications.
This code forms the backbone of Aleo network,
which verifies transactions and stores the encrypted state applications in a publicly-verifiable manner.

2. Build Guide Copy Chapter

2.1 Definitions Copy Section

The following snarkOS node types exist in the Aleo network:

• Validator: Validator nodes participate in consensus and must be started with an account that is bonded into the committee.
• Client: Clients do not participate in consensus but maintain a ledger. They are capable of providing information about the network as well as accepting solutions and transactions and communicating them to their peers. All clients run the same software, however, for the purposes of configuration management, this document defines two types of clients:
  • Core Client: Client node connected directly to a validator node.
  • Outer Client: Client node connected only to other clients or prover nodes.
• Prover: Prover nodes are dedicated to solving the Aleo puzzle. They do not participate in consensus or maintain a copy of the ledger.

2.2 Requirements Copy Section

The following are the requirements to run an Aleo node:

• OS: 64-bit architectures only, latest up-to-date for security
  • Clients: Ubuntu 22.04 (LTS), macOS Ventura or later, Windows 11 or later
  • Validators: Ubuntu 22.04 (LTS)
• CPU: 64-bit architectures only, Latest Intel Xeon or Better
  • Clients: 24-cores (32-cores or larger preferred)
  • Validators: 64-cores (128-cores or larger preferred)
• RAM: DDR4 or better
  • Clients: 128GiB of memory (192GiB or larger preferred)
  • Validators: 256GiB of memory (384GiB or larger preferred)
• Storage: PCIe Gen 3 x4, PCIe Gen 4 x2 NVME SSD, or better
  • Clients: 2TB of disk space (4TB or larger preferred)
  • Validators: 4TB of disk space (6TB or larger preferred)
• Network: Symmetric, commercial, always-on
  • Clients: 250Mbps of upload and download bandwidth
  • Validators: 500Mbps of upload and download bandwidth

No explicit recommendations are made for proving nodes as proving hardware may be highly variable. If interested in running Aleo Provers nodes, please refer to resources published by the Aleo community.

2.3 Installation Copy Section

Before beginning, please ensure your machine has Rust installed, with at least this version. Instructions to install Rust can be found here.

Start by cloning this GitHub repository:

git clone --branch mainnet --single-branch https://github.com/ProvableHQ/snarkOS.git

Next, move into the snarkOS directory:

cd snarkOS

[For Ubuntu users] A helper script to install dependencies is available. From the snarkOS directory, run:

./build_ubuntu.sh

Lastly, install snarkOS:

cargo install --locked --path .

Optional: CUDA Acceleration for Provers

This CUDA build is optional. The current snarkOS puzzle does not leverage CUDA acceleration—it is a leftover from a previous event, although CUDA may become relevant again with ARC-43.

The CUDA feature is considered unstable and experimental; expect breaking changes.

For prover operators who want to experiment with GPU support:

cargo install --locked --path . --features cuda

Please ensure ports 4130/tcp and 3030/tcp are open on your router and OS firewall.

2.4 Port Configuration Copy Section

2.4.1 For Core Clients

2.4.2 For Outer Clients

2.4.3 For Validators

Note: Ensure that your open file limit is set to 16,384 or above.
For the recommended setting run:

# Increase the open file limit for the current user (replace  with your username)
echo " - nofile 65536" | sudo tee -a /etc/security/limits.conf
# Increase the default system open file limit
sudo bash -c 'echo "DefaultLimitNOFILE=65536" >> /etc/systemd/system.conf'

3. Run an Aleo Node Copy Chapter

3.1 Run an Aleo Client Copy Chapter

Start by following the instructions in the Build Guide.
The guide below provides information on running core and outer clients (as defined in Section 2.2.) Aleo community members running validators are recommended to run 1-3 core clients as their exclusive client peers. This will ensure network traffic from the public internet is verified prior to reaching the validator.

Any client not connected directly to a validator can be considered an outer client.

3.1.1 Run an Aleo Core Client Copy Section

The following command is recommended when starting a client node that is connected to a validator:
snarkos start --client --nodisplay --node 0.0.0.0:4130 --peers "validator_ip:4130,core_client_ip_1:4130,core_client_ip_2:4130,core_client_ip3:4130,outer_client_ip_1:4130,..." --verbosity 1 --norest

To start a core client node, you can also run the following command from the snarkOS directory:

./scripts/run-core-client.sh

3.1.2 Run an Aleo Outer Client Copy Section

The following command is recommended when starting a client node that is NOT connected to a validator:
snarkos start --client --nodisplay --node 0.0.0.0:4130 --peers "core_client_ip_1:4130,core_client_ip_2:4130,core_client_ip3:4130,outer_client_ip_1:4130,..." --verbosity 1 --rest 0.0.0.0:3030

To start an outer client node, you can also run the following command from the snarkOS directory:

./scripts/run-outer-client.sh

Outer clients can be bootstrap clients that serve as accessible entry points for new nodes joining the network with publicly known or static IPs.
For bootstrap clients, we also recommend the use of --rotate-external-peers to avoid the bootstrap peerlist from filling up.

3.2 Run an Aleo Validator Copy Chapter

Start by following the instructions in the Build Guide.

The following command is recommended when starting a validator node:
snarkos start --validator --nodisplay --bft 0.0.0.0:5000 --node 0.0.0.0:4130 --peers "validator_ip_1:4130,validator_ip_2:4130,...,core_client_ip_1:4130,core_client_ip_2:4130,..." --validators "validator_ip_1:5000,validator_ip_2:5000,..." --verbosity 1 --norest --private-key-file ~/snarkOS/privatekey

Instead of specifying a private key file (--private-key-file flag), the private key can also be defined explicitly (--private-key flag).

To start a validator, you can also run the following command from the snarkOS directory:

./scripts/run-validator.sh

3.2.1 Validator Telemetry Metrics Copy Section

Validator telemetry allows you to track participation in consensus. This is enabled by default in the standard build.

Once enabled, telemetry metrics are available through:

1. Node logs
2. REST API endpoints
   text Copy

   // GET /{network}/validators/participation
   // GET /{network}/validators/participation?metadata={true}

Telemetry is enabled by default, and you can still explicitly specify the telemetry feature flag:

1. [Installation](#2.3-installation)

Use the default installation command:

cargo install --locked --path .

2. `./run-validator.sh`

Telemetry is included by default when running ./scripts/run-validator.sh.

3.3 Run an Aleo Prover Copy Chapter

Start by following the instructions in the Build Guide.

Next, generate an Aleo account address:

snarkos account new

This will output a new Aleo account in the terminal.

Please remember to save the account private key and view key. The following is an example output:

Attention - Remember to store this account private key and view key.

  Private Key  APrivateKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  <-- Save Me And Use In The Next Step
     View Key  AViewKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  <-- Save Me
      Address  aleo1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  <-- Save Me

Next, to start a proving node, from the snarkOS directory, run:

./scripts/run-prover.sh

When prompted, enter your Aleo private key:

Enter the Aleo Prover account private key:
APrivateKey1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

4. FAQs Copy Chapter

1. My node is unable to compile. Copy Section

• Ensure your machine has Rust installed, with at least this version. Instructions to install Rust can be found here.
• If large errors appear during compilation, try running cargo clean.
• Ensure snarkOS is started using ./scripts/run-client.sh or ./scripts/run-prover.sh.

2. My node is unable to connect to peers on the network. Copy Section

• Ensure ports 4130/tcp and 3030/tcp are open on your router and OS firewall.
• Ensure snarkOS is started using ./scripts/run-client.sh or ./scripts/run-prover.sh.

3. I can't generate a new address Copy Section

• Before running the command above (snarkos account new) try source ~/.bashrc
• Also double-check the spelling of snarkos. Note the directory is /snarkOS, and the command is snarkos

4. How do I use the CLI to sign and verify a message? Copy Section

1. Generate an account with snarkos account new if you haven't already
2. Sign a message with your private key using snarkos account sign --raw -m "Message" --private-key-file=
3. Verify your signature with snarkos account verify --raw -m "Message" -s sign1SignatureHere -a aleo1YourAccountAddress

Note, using the --raw flag with the command will sign plaintext messages as bytes rather than Aleo values such as 1u8 or 100field.

5. What is the different between `node-data` and `ledger`? Copy Section

Ledger contains only public ledger information, while node-data contains information specific to the node that created it. The latter should not be shared with untrusted sources, and, for validators, contains data required to participate in consensus.

6. At startup I get an error telling me the nodes still uses the old storage format. What should I do? Copy Section

The node should have created a new folder for the node data. For example, for mainnet a folder should exist at ~/.aleo/storage/node-data-0.

The error message will tell you what data to migrate. Alternatively, you can start the node with --auto-migrate-node-data and it will attempt to do this automatically.

The following is an overview of all files that may be needed to be migrated.

• The JSON Web token secret, located at ~/.aleo/storage/jwt_secrect_{address}.txt. Move it to ~/.aleo/storage/node-data-0/jwt_secret_{address}.txt.
  Note, if you are running different nodes with different addresses there may be multiple of these. The error message will tell you which one to migrate.
• The router peer cache located at ~/.aleo/storage/ledger-0/cache_router_peers. Migrate it to ~/.aleo/storage/node-data-0/router-peer-cache.
• The gateway peer cache located at ~/.aleo/storage/ledger-0/cache_gateway_peers (only for validators). Migrate it to ~/.aleo/storage/node-data-0/gateway-peer-cache.
• The latest proposal cache located at ~/.aleo/storage/current-proposal-cache-0 (only for validators). Migrate it to ~/.aleo/storage/node-data-0/current-proposal-cache.

5. Command Line Interface Copy Chapter

To run a node with custom settings, refer to the options and flags available in the snarkOS CLI.

The full list of CLI flags and options can be viewed with snarkos --help:

snarkOS 
The Aleo Team <[email protected]>

USAGE:
    snarkos [OPTIONS] <SUBCOMMAND>

OPTIONS:
    -h, --help                     Print help information
    -v, --verbosity <VERBOSITY>    Specify the verbosity [options: 0, 1, 2, 3] [default: 2]

SUBCOMMANDS:
    account    Commands to manage Aleo accounts
    clean      Cleans the snarkOS node storage
    help       Print this message or the help of the given subcommand(s)
    start      Starts the snarkOS node
    update     Update snarkOS

The following are the options for the snarkos start command:

/* Detailed source-code truncated for AI context efficiency. */

6. Development Guide Copy Chapter

6.1 Quick Start Copy Section

In the first terminal, start the first validator by running:

cargo run --release -- start --nodisplay --dev 0 --validator

In the second terminal, start the second validator by running:

cargo run --release -- start --nodisplay --dev 1 --validator

In the third terminal, start the third validator by running:

cargo run --release -- start --nodisplay --dev 2 --validator

In the fourth terminal, start the fourth validator by running:

cargo run --release -- start --nodisplay --dev 3 --validator

From here, this procedure can be used to further start-up provers and clients.

6.2 Operations Copy Section

It is important to initialize the nodes starting from 0 and incrementing by 1 for each new node.

The following is a list of options to initialize a node (replace <NODE_ID> with a number starting from 0):

cargo run --release -- start --nodisplay --dev <NODE_ID> --validator
cargo run --release -- start --nodisplay --dev <NODE_ID> --prover
cargo run --release -- start --nodisplay --dev <NODE_ID> --client
cargo run --release -- start --nodisplay --dev <NODE_ID>

When no node type is specified, the node will default to --client.

6.3 Local Devnet Copy Section

6.3.1 Install `tmux`

To run a local devnet with the script, start by installing tmux.

macOS

To install tmux on macOS, you can use the Homebrew package manager.
If you haven't installed Homebrew yet, you can find instructions at their website.

# Once Homebrew is installed, run:
brew install tmux

Ubuntu

On Ubuntu and other Debian-based systems, you can use the apt package manager:

sudo apt update
sudo apt install tmux

Windows

There are a couple of ways to use tmux on Windows:

Using Windows Subsystem for Linux (WSL) Copy Section

1. First, install Windows Subsystem for Linux.
2. Once WSL is set up and you have a Linux distribution installed (e.g., Ubuntu), open your WSL terminal and install tmux as you would on a native Linux system:

sudo apt update
sudo apt install tmux

6.3.2 Start a Local Devnet

To start a local devnet, run:

./scripts/devnet.sh

Follow the instructions in the terminal to start the devnet.

6.3.3 View a Local Devnet

Switch Nodes (forward)

To toggle to the next node in a local devnet, run:

Ctrl+b n

Switch Nodes (backwards)

To toggle to the previous node in a local devnet, run:

Ctrl+b p

Select a Node (choose-tree)

To select a node in a local devnet, run:

Ctrl+b w

Select a Node (manually)

To select a node manually in a local devnet, run:

Ctrl+b :select-window -t {NODE_ID}

6.3.4 Stop a Local Devnet

To stop a local devnet, run:

Ctrl+b :kill-session

Then, press Enter.

Clean Up Copy Section

To clean up the node storage, run:

cargo run --release -- clean --dev <NODE_ID>

6.4 Feature Flags Copy Chapter

By default, the metrics and telemetry features are turned on for some internal crates.

• history -
  Enables a /history REST endpoint.
• telemetry -
  Allows the node to upload telemetry data.
• cuda -
  Allows some operations to run on the (NVidia) GPU, instead of on the CPU. See CUDA acceleration for provers for install tips and current puzzle status.
• locktick -
  This feature turns on code for detecting deadlocks.
• test_targets -
  This feature allows the lowering of coinbase and proof targets for testing.

6.5 Local Backups Copy Chapter

The snarkOS node implementation uses rocksdb under the hood. By using its native checkpointing mechanism, you can create backups locally and efficiently. The backups leverage hard links on your filesystem, thereby incurring only a marginal amount of extra space. The aim of these local backups is for you to be able to recover quickly in case your node were to halt.

Note: in order for the backups to be incremental and lightweight, they need to be stored at the same filesystem (this includes btrfs subvolumes) as the ledger; otherwise, they become full copies.

You can find a basic sample script in scripts/backup.sh which you can run as a cron-job e.g. every minute. Each run of the script creates a new backup folder with a timestamp postfix. It will ensure a backup is kept which is 1 minute old, 5 minutes old, 1 hour old and 1 day old. In more detail, on each run it will:

• always overwrite the latest backup
• only overwrite the 5 minute backup if it is older than 5 minutes
• only overwrite the 1 hour backup if it is older than 1 hour
• only overwrite the 1 day backup if it is older than 1 day

You may want to change the NETWORK, BASE_DIR, ENDPOINT and JWT variables.

7. Contributors Copy Chapter

Thank you for helping make snarkOS better!
🧐 What do the emojis mean?

This project follows the all-contributors specification. Contributions of any kind are welcome!

8. License Copy Chapter

We welcome all contributions to snarkOS. Please refer to the license for the terms of contributions.