rathole Documentation

Technical Documentation: rathole-org/rathole

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

rathole

English | 简体中文

A secure, stable and high-performance reverse proxy for NAT traversal, written in Rust

rathole, like frp and ngrok, can help to expose the service on the device behind the NAT to the Internet, via a server with a public IP.

• rathole
  • Features
  • Quickstart
  • Configuration
    • Logging
    • Tuning
  • Benchmark
  • Planning

Features Copy Chapter

• High Performance Much higher throughput can be achieved than frp, and more stable when handling a large volume of connections. See Benchmark
• Low Resource Consumption Consumes much fewer memory than similar tools. See Benchmark. The binary can be as small as ~500KiB to fit the constraints of devices, like embedded devices as routers.
• Security Tokens of services are mandatory and service-wise. The server and clients are responsible for their own configs. With the optional Noise Protocol, encryption can be configured at ease. No need to create a self-signed certificate! TLS is also supported.
• Hot Reload Services can be added or removed dynamically by hot-reloading the configuration file. HTTP API is WIP.

Quickstart Copy Chapter

A full-powered rathole can be obtained from the release page. Or build from source for other platforms and minimizing the binary. A Docker image is also available.

The usage of rathole is very similar to frp. If you have experience with the latter, then the configuration is very easy for you. The only difference is that configuration of a service is split into the client side and the server side, and a token is mandatory.

To use rathole, you need a server with a public IP, and a device behind the NAT, where some services that need to be exposed to the Internet.

Assuming you have a NAS at home behind the NAT, and want to expose its ssh service to the Internet:

1. On the server which has a public IP

Create server.toml with the following content and accommodate it to your needs.

# server.toml
[server]
bind_addr = "0.0.0.0:2333" # `2333` specifies the port that rathole listens for clients

[server.services.my_nas_ssh]
token = "use_a_secret_that_only_you_know" # Token that is used to authenticate the client for the service. Change to an arbitrary value.
bind_addr = "0.0.0.0:5202" # `5202` specifies the port that exposes `my_nas_ssh` to the Internet

Then run:

./rathole server.toml

2. On the host which is behind the NAT (your NAS)

Create client.toml with the following content and accommodate it to your needs.

# client.toml
[client]
remote_addr = "myserver.com:2333" # The address of the server. The port must be the same with the port in `server.bind_addr`

[client.services.my_nas_ssh]
token = "use_a_secret_that_only_you_know" # Must be the same with the server to pass the validation
local_addr = "127.0.0.1:22" # The address of the service that needs to be forwarded

Then run:

./rathole client.toml

3. Now the client will try to connect to the server myserver.com on port 2333, and any traffic to myserver.com:5202 will be forwarded to the client's port 22.

So you can ssh myserver.com:5202 to ssh to your NAS.

To run rathole run as a background service on Linux, checkout the systemd examples.

Configuration Copy Chapter

rathole can automatically determine to run in the server mode or the client mode, according to the content of the configuration file, if only one of [server] and [client] block is present, like the example in Quickstart.

But the [client] and [server] block can also be put in one file. Then on the server side, run rathole --server config.toml and on the client side, run rathole --client config.toml to explicitly tell rathole the running mode.

Before heading to the full configuration specification, it's recommend to skim the configuration examples to get a feeling of the configuration format.

See Transport for more details about encryption and the transport block.

Here is the full configuration specification:

/* Detailed source-code truncated for AI context efficiency. */

Logging Copy Section

rathole, like many other Rust programs, use environment variables to control the logging level. info, warn, error, debug, trace are available.

RUST_LOG=error ./rathole config.toml

will run rathole with only error level logging.

If RUST_LOG is not present, the default logging level is info.

Tuning Copy Section

From v0.4.7, rathole enables TCP_NODELAY by default, which should benefit the latency and interactive applications like rdp, Minecraft servers. However, it slightly decreases the bandwidth.

If the bandwidth is more important, TCP_NODELAY can be opted out with nodelay = false.

Benchmark Copy Chapter

rathole has similar latency to frp, but can handle a more connections, provide larger bandwidth, with less memory usage.

For more details, see the separate page Benchmark.

However, don't take it from here that rathole can magically make your forwarded service faster several times than before. The benchmark is done on local loopback, indicating the performance when the task is cpu-bounded. One can gain quite a improvement if the network is not the bottleneck. Unfortunately, that's not true for many users. In that case, the main benefit is lower resource consumption, while the bandwidth and the latency may not improved significantly.

Planning Copy Chapter

• HTTP APIs for configuration

Out of Scope lists features that are not planned to be implemented and why.

Overview (`examples/systemd/README.md`) Copy Section

Systemd Unit Examples Copy Chapter

The directory lists some systemd unit files for example, which can be used to run rathole as a service on Linux.

The @ symbol in the name of unit files such as
[email protected] facilitates the management of multiple instances of rathole.

For the naming of the example, ratholes stands for rathole --server, and ratholec stands for rathole --client, rathole is just rathole.

For security, it is suggested to store configuration files with permission 600, that is, only the owner can read the file, preventing arbitrary users on the system from accessing the secret tokens.

With root privilege Copy Section

Assuming that rathole is installed in /usr/bin/rathole, and the configuration file is in /etc/rathole/app1.toml, the following steps show how to run an instance of rathole --server with root.

1. Create a service file.

sudo cp [email protected] /etc/systemd/system/

2. Create the configuration file app1.toml.

sudo mkdir -p /etc/rathole
# And create the configuration file named `app1.toml` inside /etc/rathole

3. Enable and start the service.

sudo systemctl daemon-reload # Make sure systemd find the new unit
sudo systemctl enable ratholes@app1 --now

Without root privilege Copy Section

Assuming that rathole is installed in ~/.local/bin/rathole, and the configuration file is in ~/.local/etc/rathole/app1.toml, the following steps show how to run an instance of rathole --server without root.

1. Edit the example service file as...

# with root
# ExecStart=/usr/bin/rathole -s /etc/rathole/%i.toml
# without root
ExecStart=%h/.local/bin/rathole -s %h/.local/etc/rathole/%i.toml

2. Create a service file.

mkdir -p ~/.config/systemd/user
cp [email protected] ~/.config/systemd/user/

3. Create the configuration file app1.toml.

mkdir -p ~/.local/etc/rathole
# And create the configuration file named `app1.toml` inside ~/.local/etc/rathole

4. Enable and start the service.

systemctl --user daemon-reload # Make sure systemd find the new unit
systemctl --user enable ratholes@app1 --now

Run multiple services Copy Section

To run multiple services at once, simply add another configuration, say app2.toml under /etc/rathole (~/.local/etc/rathole for non-root), then run sudo systemctl enable ratholes@app2 --now (systemctl --user enable ratholes@app2 --now for non-root) to start an instance for that configuration.

The same applies to [email protected] for rathole --client and [email protected] for rathole.

Benchmark (`docs/benchmark.md`) Copy Section

Benchmark

Date: 2021/12/28

Version: commit 1180c7e538564efd69742f22e77453a1b74a5ed2

Arch Linux with 5.15.11-arch2-1 kernel

Intel Xeon CPU E5-2620 @ 2.00GHz *2

16GB RAM

Bandwidth Copy Chapter

rathole with the following configuration:

[client]
remote_addr = "localhost:2333"
default_token = "123"

[client.services.bench-tcp]
local_addr = "127.0.0.1:80"
[client.services.bench-udp]
type = "udp"
local_addr = "127.0.0.1:80"

[server]
bind_addr = "0.0.0.0:2333"
default_token = "123"

[server.services.bench-tcp]
bind_addr = "0.0.0.0:5202"
[server.services.bench-udp]
type = "udp"
bind_addr = "0.0.0.0:5202"

frp 0.38.0 with the following configuration:

[common]
bind_port = 7000
authentication_method = token
token = 1233

# frpc.ini
[common]
server_addr = 127.0.0.1
server_port = 7000
authentication_method = token
token = 1233

[bench-tcp]
type = tcp
local_ip = 127.0.0.1
local_port = 80
remote_port = 5203
[bench-udp]
type = udp
local_ip = 127.0.0.1
local_port = 80
remote_port = 5203

$ iperf3 -v
iperf 3.10.1 (cJSON 1.7.13)
Linux sig 5.15.7-arch1-1 #1 SMP PREEMPT Wed, 08 Dec 2021 14:33:16 +0000 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment
$ sudo iperf3 -s -p 80

For rathole benchmark:

$ iperf3 -c 127.0.0.1 -p 5202

For frp benchmark:

$ iperf3 -c 127.0.0.1 -p 5203

HTTP Copy Chapter

nginx/1.20.2 listens on port 80, with the default test page.

frp and rathole configuration is same with the previous section.

vegeta is used to generate HTTP load.

HTTP Throughput Copy Section

The following commands are used to benchmark rathole and frp. Note that if you want to do a benchmark yourself, -max-workers should be adjusted to get the accurate results for your machine.

echo 'GET http://127.0.0.1:5203' | vegeta attack -rate 0 -duration 30s -max-workers 48
echo 'GET http://127.0.0.1:5202' | vegeta attack -rate 0 -duration 30s -max-workers 48

HTTP Latency Copy Section

rathole has very similar latency to frp, but can handle more connections

Here's a table, latency is in ms

As you can see, for QPS from 1 to 3000, rathole and frp have nearly identical latency.
But with QPS of 4000, frp starts reporting lots of errors and the latency grows to even seconds. This kind of reflects the throughput in the previous section.

Thus, in terms of latency, rathole and frp are nearly the same. But rathole can handle more connections.

Script to benchmark latency

Memory Usage Copy Chapter

The graph shows the memory usage of frp and rathole when vegeta attack -duration 30s -rate 1000 is executed.
rathole uses much less memory than frp.

Script to benchmark memory

Build Guide (`docs/build-guide.md`) Copy Section

Build Guide

This is for those who want to build rathole themselves, possibly because the need of latest features or the minimal binary size.

Build Copy Chapter

To use default build settings, run:

cargo build --release

You may need to pre-install openssl dependencies in Unix-like systems.

Customize the Build Copy Chapter

rathole comes with lots of crate features that determine whether a certain feature will be compiled or not. Supported features can be checked out in [features] of Cargo.toml.

For example, to build rathole with the client and noise feature:

cargo build --release --no-default-features --features client,noise

Rustls Support Copy Chapter

rathole provides optional rustls support. It's an almost drop-in replacement of native-tls support. (See Transport for more information.)

To enable this, disable the default features and enable rustls feature. And for websocket feature, enable websocket-rustls feature as well.

You can also use command line option for this. For example, to replace all default features with rustls:

cargo build --release --no-default-features --features server,client,rustls,noise,websocket-rustls,hot-reload

Feature rustls and websocket-rustls cannot be enabled with native-tls and websocket-native-tls at the same time, as they are mutually exclusive. Enabling both will result in a compile error.

(Note that default features contains native-tls and websocket-native-tls.)

Minimalize the binary Copy Chapter

1. Build with the minimal profile

The release build profile optimize for the program running time, not the binary size.

However, the minimal profile enables lots of optimization for the binary size to produce a much smaller binary.

For example, to build rathole with client feature with the minimal profile:

cargo build --profile minimal --no-default-features --features client

2. strip and upx

The binary that step 1 produces can be even smaller, by using strip and upx to remove the symbols and compress the binary.

Like:

strip rathole
upx --best --lzma rathole

At the time of writting the build guide, the produced binary for x86_64-unknown-linux-glibc has the size of 574 KiB, while frpc has the size of ~10 MiB, which is much larger.

Internals (`docs/internals.md`) Copy Section

Internals

Conceptions Copy Chapter

Service Copy Section

The entity whose traffic needs to be forwarded

Server Copy Section

The host that runs rathole in the server mode

Client Copy Section

The host behind the NAT that runs rathole in the client mode. It has some services that need to be forwarded.

Visitor Copy Section

Who visists a service, via the server

Control Channel Copy Section

A control channel is a TCP connection between the server and the client that only carries rathole control commands for one service.

Data Channel Copy Section

A data channel is a TCP connection between the server and the client that only carries the encapsulated data that needs forwarding for one service.

The Process Copy Chapter

TODO: Add more details about the protocol

When rathole starts in the client mode, it creates connections to server.common.bind_addr for each service. These connection acts as control channels.

When a control channel starts, the server challenge the client by a nonce, the client is required to authenticate as the service it wants to represent. Then the forwarding of that service is set up.

When the server accepts a connection on a service's bind_port, it sends a control command to the client via the corresponding control channel. Then the client connects to the server to create a data channel. In this way, a forwarding is set up. The server also creates a few data channels in advance to improve the latency.

Out Of Scope (`docs/out-of-scope.md`) Copy Section

Out of Scope

rathole focuses on the forwarding for the NAT traversal, rather than being a all-in-one development tool or a load balancer or a gateway. It's designed to be used with them, not replace them.

But that doesn't mean it's not useful for other purposes. In the future, more configuration APIs will be added and rathole can be used with an external dashboard.

Make each program do one thing well.

• Domain based forwarding for HTTP

  Introducing these kind of features into rathole itself ultimately reinvent a nginx. Use nginx to do this and set rathole as the upstream. This method achieves better performance as well as flexibility.

• HTTP Request Logging

  rathole doesn't interference with the application layer traffic. A right place for this kind of stuff is the web server, and a network capture tool.

• frp's STCP or other setup that requires visitors' side configuration

  If that kind of setup is possible, then there are a lot more tools available. You may want to consider secure tunnels like wireguard or zerotier. rathole primarily focuses on NAT traversal by forwarding, which doesn't require any setup for visitors.

• Caching local_ip's DNS records

  As responded in issue #183, local_ip cache is not feasible because we have no reliable way to detect ip change. Handle DNS TTL and so on should be done with a DNS server, not a client. Caching ip is generally dangerous for clients. If you care about the local_ip query you can set up a local DNS server and enable caching. Then the local lookup should be trivial.

Transport (`docs/transport.md`) Copy Section

Security

By default, rathole forwards traffic as it is. Different options can be enabled to secure the traffic.

TLS Copy Chapter

Checkout the example

Client Copy Section

Normally, a self-signed certificate is used. In this case, the client needs to trust the CA. trusted_root is the path to the root CA's certificate PEM file.
hostname is the hostname that the client used to validate aginst the certificate that the server presents. Note that it does not have to be the same with the remote_addr in [client].

[client.transport.tls]
trusted_root = "example/tls/rootCA.crt"
hostname = "localhost"

Server Copy Section

PKCS#12 archives are needed to run the server.

It can be created using openssl like:

openssl pkcs12 -export -out identity.pfx -inkey server.key -in server.crt -certfile ca_chain_certs.crt

Aruguments are:

• -inkey: Server Private Key
• -in: Server Certificate
• -certfile: CA Certificate

Creating self-signed certificate with one's own CA is a non-trival task. However, a script is provided under tls example folder for reference.

Rustls Support Copy Section

rathole provides optional rustls support. Build Guide demostrated this.

One difference is that, the crate we use for loading PKCS#12 archives can only handle limited types of PBE algorithms. We only support PKCS#12 archives that they (crate p12) support. So we need to specify the legacy format (openssl 1.x format) when creating the PKCS#12 archive.

In short, the command used with openssl 3 to create the PKCS#12 archive with rustls support is:

openssl pkcs12 -export -out identity.pfx -inkey server.key -in server.crt -certfile ca_chain_certs.crt -legacy

Noise Protocol Copy Chapter

Quickstart for the Noise Protocl Copy Section

In one word, the Noise Protocol is a lightweigt, easy to configure and drop-in replacement of TLS. No need to create a self-sign certificate to secure the connection.

rathole comes with a reasonable default configuration for noise protocol. You can a glimpse of the minimal example for how it will look like.

The default noise protocol that rathole uses, which is Noise_NK_25519_ChaChaPoly_BLAKE2s, providing the authentication of the server, just like TLS with properly configured certificates. So MITM is no more a problem.

To use it, a X25519 keypair is needed.

Generate a Keypair

1. Run rathole --genkey, which will generate a keypair using the default X25519 algorithm.

It emits:

$ rathole --genkey
Private Key:
cQ/vwIqNPJZmuM/OikglzBo/+jlYGrOt9i0k5h5vn1Q=

Public Key:
GQYTKSbWLBUSZiGfdWPSgek9yoOuaiwGD/GIX8Z1kkE=

(WARNING: Don't use the keypair from the Internet, including this one)

2. The server should keep the private key to identify itself. And the client should keep the public key, which is used to verify whether the peer is the authentic server.

So relevant snippets of configuration are:

# Client Side Configuration
[client.transport]
type = "noise"
[client.transport.noise]
remote_public_key = "GQYTKSbWLBUSZiGfdWPSgek9yoOuaiwGD/GIX8Z1kkE="

# Server Side Configuration
[server.transport]
type = "noise"
[server.transport.noise]
local_private_key = "cQ/vwIqNPJZmuM/OikglzBo/+jlYGrOt9i0k5h5vn1Q="

Then rathole will run under the protection of the Noise Protocol.

Specifying the Pattern of Noise Protocol Copy Chapter

The default configuration of Noise Protocol that comes with rathole satifies most use cases, which is described above. But there're other patterns that can be useful.

No Authentication Copy Section

This configuration provides encryption of the traffic but provides no authentication, which means it's vulnerable to MITM attack, but is resistent to the sniffing and replay attack. If MITM attack is not one of the concerns, this is more convenient to use.

# Server Side Configuration
[server.transport.noise]
pattern = "Noise_XX_25519_ChaChaPoly_BLAKE2s"

# Client Side Configuration
[client.transport.noise]
pattern = "Noise_XX_25519_ChaChaPoly_BLAKE2s"

Bidirectional Authentication Copy Section

# Server Side Configuration
[server.transport.noise]
pattern = "Noise_KK_25519_ChaChaPoly_BLAKE2s"
local_private_key = "server-priv-key-here"
remote_public_key = "client-pub-key-here"

# Client Side Configuration
[client.transport.noise]
pattern = "Noise_KK_25519_ChaChaPoly_BLAKE2s"
local_private_key = "client-priv-key-here"
remote_public_key = "server-pub-key-here"

Other Patterns Copy Section

To find out which pattern to use, refer to:

• 7.5. Interactive handshake patterns (fundamental)
• 8. Protocol names and modifiers

Note that PSKs are not supported currently. Free to open an issue if you need it.

--- METRICS ---

• Files Extracted: 7
• Estimated Token Budget: ~5658 tokens
• Canonical Reference: https://codewiki.google/github.com/rathole-org/rathole