# Technical Documentation: rathole-org/rathole > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/rathole-org/rathole) ### Overview (`README.md`) # rathole [](https://github.com/rapiz1/rathole/stargazers) [](https://github.com/rapiz1/rathole/releases) [](https://github.com/rapiz1/rathole/releases) [](https://hub.docker.com/r/rapiz1/rathole) [](https://gitter.im/rapiz1/rathole?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [English](README.md) | [简体中文](README-zh.md) A secure, stable and high-performance reverse proxy for NAT traversal, written in Rust rathole, like [frp](https://github.com/fatedier/frp) and [ngrok](https://github.com/inconshreveable/ngrok), can help to expose the service on the device behind the NAT to the Internet, via a server with a public IP. - [rathole](#rathole) - [Features](#features) - [Quickstart](#quickstart) - [Configuration](#configuration) - [Logging](#logging) - [Tuning](#tuning) - [Benchmark](#benchmark) - [Planning](#planning) ## Features - **High Performance** Much higher throughput can be achieved than frp, and more stable when handling a large volume of connections. See [Benchmark](#benchmark) - **Low Resource Consumption** Consumes much fewer memory than similar tools. See [Benchmark](#benchmark). [The binary can be](docs/build-guide.md) **as small as ~500KiB** to fit the constraints of devices, like embedded devices as routers. - **Security** Tokens of services are mandatory and service-wise. The server and clients are responsible for their own configs. With the optional Noise Protocol, encryption can be configured at ease. No need to create a self-signed certificate! TLS is also supported. - **Hot Reload** Services can be added or removed dynamically by hot-reloading the configuration file. HTTP API is WIP. ## Quickstart A full-powered `rathole` can be obtained from the [release](https://github.com/rapiz1/rathole/releases) page. Or [build from source](docs/build-guide.md) **for other platforms and minimizing the binary**. A [Docker image](https://hub.docker.com/r/rapiz1/rathole) is also available. The usage of `rathole` is very similar to frp. If you have experience with the latter, then the configuration is very easy for you. The only difference is that configuration of a service is split into the client side and the server side, and a token is mandatory. To use `rathole`, you need a server with a public IP, and a device behind the NAT, where some services that need to be exposed to the Internet. Assuming you have a NAS at home behind the NAT, and want to expose its ssh service to the Internet: 1. On the server which has a public IP Create `server.toml` with the following content and accommodate it to your needs. ```toml # server.toml [server] bind_addr = "0.0.0.0:2333" # `2333` specifies the port that rathole listens for clients [server.services.my_nas_ssh] token = "use_a_secret_that_only_you_know" # Token that is used to authenticate the client for the service. Change to an arbitrary value. bind_addr = "0.0.0.0:5202" # `5202` specifies the port that exposes `my_nas_ssh` to the Internet ``` Then run: ```bash ./rathole server.toml ``` 2. On the host which is behind the NAT (your NAS) Create `client.toml` with the following content and accommodate it to your needs. ```toml # client.toml [client] remote_addr = "myserver.com:2333" # The address of the server. The port must be the same with the port in `server.bind_addr` [client.services.my_nas_ssh] token = "use_a_secret_that_only_you_know" # Must be the same with the server to pass the validation local_addr = "127.0.0.1:22" # The address of the service that needs to be forwarded ``` Then run: ```bash ./rathole client.toml ``` 3. Now the client will try to connect to the server `myserver.com` on port `2333`, and any traffic to `myserver.com:5202` will be forwarded to the client's port `22`. So you can `ssh myserver.com:5202` to ssh to your NAS. To run `rathole` run as a background service on Linux, checkout the [systemd examples](./examples/systemd). ## Configuration `rathole` can automatically determine to run in the server mode or the client mode, according to the content of the configuration file, if only one of `[server]` and `[client]` block is present, like the example in [Quickstart](#quickstart). But the `[client]` and `[server]` block can also be put in one file. Then on the server side, run `rathole --server config.toml` and on the client side, run `rathole --client config.toml` to explicitly tell `rathole` the running mode. Before heading to the full configuration specification, it's recommend to skim [the configuration examples](./examples) to get a feeling of the configuration format. See [Transport](./docs/transport.md) for more details about encryption and the `transport` block. Here is the full configuration specification: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Logging `rathole`, like many other Rust programs, use environment variables to control the logging level. `info`, `warn`, `error`, `debug`, `trace` are available. ```shell RUST_LOG=error ./rathole config.toml ``` will run `rathole` with only error level logging. If `RUST_LOG` is not present, the default logging level is `info`. ### Tuning From v0.4.7, rathole enables TCP_NODELAY by default, which should benefit the latency and interactive applications like rdp, Minecraft servers. However, it slightly decreases the bandwidth. If the bandwidth is more important, TCP_NODELAY can be opted out with `nodelay = false`. ## Benchmark rathole has similar latency to [frp](https://github.com/fatedier/frp), but can handle a more connections, provide larger bandwidth, with less memory usage. For more details, see the separate page [Benchmark](./docs/benchmark.md). **However, don't take it from here that `rathole` can magically make your forwarded service faster several times than before.** The benchmark is done on local loopback, indicating the performance when the task is cpu-bounded. One can gain quite a improvement if the network is not the bottleneck. Unfortunately, that's not true for many users. In that case, the main benefit is lower resource consumption, while the bandwidth and the latency may not improved significantly. ## Planning - [ ] HTTP APIs for configuration [Out of Scope](./docs/out-of-scope.md) lists features that are not planned to be implemented and why. --- ### Overview (`examples/systemd/README.md`) ## Systemd Unit Examples The directory lists some systemd unit files for example, which can be used to run `rathole` as a service on Linux. [The `@` symbol in the name of unit files](https://superuser.com/questions/393423/the-symbol-and-systemctl-and-vsftpd) such as `rathole@.service` facilitates the management of multiple instances of `rathole`. For the naming of the example, `ratholes` stands for `rathole --server`, and `ratholec` stands for `rathole --client`, `rathole` is just `rathole`. For security, it is suggested to store configuration files with permission `600`, that is, only the owner can read the file, preventing arbitrary users on the system from accessing the secret tokens. ### With root privilege Assuming that `rathole` is installed in `/usr/bin/rathole`, and the configuration file is in `/etc/rathole/app1.toml`, the following steps show how to run an instance of `rathole --server` with root. 1. Create a service file. ```bash sudo cp ratholes@.service /etc/systemd/system/ ``` 2. Create the configuration file `app1.toml`. ```bash sudo mkdir -p /etc/rathole # And create the configuration file named `app1.toml` inside /etc/rathole ``` 3. Enable and start the service. ```bash sudo systemctl daemon-reload # Make sure systemd find the new unit sudo systemctl enable ratholes@app1 --now ``` ### Without root privilege Assuming that `rathole` is installed in `~/.local/bin/rathole`, and the configuration file is in `~/.local/etc/rathole/app1.toml`, the following steps show how to run an instance of `rathole --server` without root. 1. Edit the example service file as... ```txt # with root # ExecStart=/usr/bin/rathole -s /etc/rathole/%i.toml # without root ExecStart=%h/.local/bin/rathole -s %h/.local/etc/rathole/%i.toml ``` 2. Create a service file. ```bash mkdir -p ~/.config/systemd/user cp ratholes@.service ~/.config/systemd/user/ ``` 3. Create the configuration file `app1.toml`. ```bash mkdir -p ~/.local/etc/rathole # And create the configuration file named `app1.toml` inside ~/.local/etc/rathole ``` 4. Enable and start the service. ```bash systemctl --user daemon-reload # Make sure systemd find the new unit systemctl --user enable ratholes@app1 --now ``` ### Run multiple services To run multiple services at once, simply add another configuration, say `app2.toml` under `/etc/rathole` (`~/.local/etc/rathole` for non-root), then run `sudo systemctl enable ratholes@app2 --now` (`systemctl --user enable ratholes@app2 --now` for non-root) to start an instance for that configuration. The same applies to `ratholec@.service` for `rathole --client` and `rathole@.service` for `rathole`. --- ### Benchmark (`docs/benchmark.md`) # Benchmark > Date: 2021/12/28 > > Version: commit 1180c7e538564efd69742f22e77453a1b74a5ed2 > > Arch Linux with 5.15.11-arch2-1 kernel > > Intel Xeon CPU E5-2620 @ 2.00GHz *2 > > 16GB RAM ## Bandwidth rathole with the following configuration: ```toml [client] remote_addr = "localhost:2333" default_token = "123" [client.services.bench-tcp] local_addr = "127.0.0.1:80" [client.services.bench-udp] type = "udp" local_addr = "127.0.0.1:80" [server] bind_addr = "0.0.0.0:2333" default_token = "123" [server.services.bench-tcp] bind_addr = "0.0.0.0:5202" [server.services.bench-udp] type = "udp" bind_addr = "0.0.0.0:5202" ``` frp 0.38.0 with the following configuration: ```ini [common] bind_port = 7000 authentication_method = token token = 1233 ``` ```ini # frpc.ini [common] server_addr = 127.0.0.1 server_port = 7000 authentication_method = token token = 1233 [bench-tcp] type = tcp local_ip = 127.0.0.1 local_port = 80 remote_port = 5203 [bench-udp] type = udp local_ip = 127.0.0.1 local_port = 80 remote_port = 5203 ``` ``` $ iperf3 -v iperf 3.10.1 (cJSON 1.7.13) Linux sig 5.15.7-arch1-1 #1 SMP PREEMPT Wed, 08 Dec 2021 14:33:16 +0000 x86_64 Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment $ sudo iperf3 -s -p 80 ``` For rathole benchmark: ``` $ iperf3 -c 127.0.0.1 -p 5202 ``` For frp benchmark: ``` $ iperf3 -c 127.0.0.1 -p 5203 ``` ## HTTP nginx/1.20.2 listens on port 80, with the default test page. frp and rathole configuration is same with the previous section. [vegeta](https://github.com/tsenart/vegeta) is used to generate HTTP load. ### HTTP Throughput The following commands are used to benchmark rathole and frp. Note that if you want to do a benchmark yourself, `-max-workers` should be adjusted to get the accurate results for your machine. ``` echo 'GET http://127.0.0.1:5203' | vegeta attack -rate 0 -duration 30s -max-workers 48 echo 'GET http://127.0.0.1:5202' | vegeta attack -rate 0 -duration 30s -max-workers 48 ``` ### HTTP Latency `rathole` has very similar latency to `frp`, but can handle more connections Here's a table, latency is in ms |QPS|latency(rathole)|latency(frp)| |--|--|---| |1|2.113|2.55| |1000|1.723|1.742| |2000|1.845|1.749| |3000|2.064|2.011| |4000|2.569|7907| As you can see, for QPS from 1 to 3000, rathole and frp have nearly identical latency. But with QPS of 4000, frp starts reporting lots of errors and the latency grows to even seconds. This kind of reflects the throughput in the previous section. Thus, in terms of latency, rathole and frp are nearly the same. But rathole can handle more connections. [Script to benchmark latency](../benches/scripts/http/latency.sh) ## Memory Usage The graph shows the memory usage of frp and rathole when `vegeta attack -duration 30s -rate 1000` is executed. rathole uses much less memory than frp. [Script to benchmark memory](../benches/scripts/mem/mem.sh) --- ### Build Guide (`docs/build-guide.md`) # Build Guide This is for those who want to build `rathole` themselves, possibly because the need of latest features or the minimal binary size. ## Build To use default build settings, run: ```sh cargo build --release ``` You may need to pre-install [openssl](https://docs.rs/openssl/latest/openssl/index.html) dependencies in Unix-like systems. ## Customize the Build `rathole` comes with lots of *crate features* that determine whether a certain feature will be compiled or not. Supported features can be checked out in `[features]` of [Cargo.toml](../Cargo.toml). For example, to build `rathole` with the `client` and `noise` feature: ```sh cargo build --release --no-default-features --features client,noise ``` ## Rustls Support `rathole` provides optional `rustls` support. It's an almost drop-in replacement of `native-tls` support. (See [Transport](transport.md) for more information.) To enable this, disable the default features and enable `rustls` feature. And for websocket feature, enable `websocket-rustls` feature as well. You can also use command line option for this. For example, to replace all default features with `rustls`: ```sh cargo build --release --no-default-features --features server,client,rustls,noise,websocket-rustls,hot-reload ``` Feature `rustls` and `websocket-rustls` cannot be enabled with `native-tls` and `websocket-native-tls` at the same time, as they are mutually exclusive. Enabling both will result in a compile error. (Note that default features contains `native-tls` and `websocket-native-tls`.) ## Minimalize the binary 1. Build with the `minimal` profile The `release` build profile optimize for the program running time, not the binary size. However, the `minimal` profile enables lots of optimization for the binary size to produce a much smaller binary. For example, to build `rathole` with `client` feature with the `minimal` profile: ```sh cargo build --profile minimal --no-default-features --features client ``` 2. `strip` and `upx` The binary that step 1 produces can be even smaller, by using `strip` and `upx` to remove the symbols and compress the binary. Like: ```sh strip rathole upx --best --lzma rathole ``` At the time of writting the build guide, the produced binary for `x86_64-unknown-linux-glibc` has the size of **574 KiB**, while `frpc` has the size of **~10 MiB**, which is much larger. --- ### Internals (`docs/internals.md`) # Internals ## Conceptions ### Service The entity whose traffic needs to be forwarded ### Server The host that runs `rathole` in the server mode ### Client The host behind the NAT that runs `rathole` in the client mode. It has some services that need to be forwarded. ### Visitor Who visists a *service*, via the *server* ### Control Channel A control channel is a TCP connection between the *server* and the *client* that only carries `rathole` control commands for one *service*. ### Data Channel A data channel is a TCP connection between the *server* and the *client* that only carries the encapsulated data that needs forwarding for one *service*. ## The Process *TODO: Add more details about the protocol* When `rathole` starts in the client mode, it creates connections to `server.common.bind_addr` for each service. These connection acts as control channels. When a control channel starts, the server challenge the client by a nonce, the client is required to authenticate as the service it wants to represent. Then the forwarding of that service is set up. When the server accepts a connection on a service's `bind_port`, it sends a control command to the client via the corresponding control channel. Then the client connects to the server to create a data channel. In this way, a forwarding is set up. The server also creates a few data channels in advance to improve the latency. --- ### Out Of Scope (`docs/out-of-scope.md`) # Out of Scope `rathole` focuses on the forwarding for the NAT traversal, rather than being a all-in-one development tool or a load balancer or a gateway. It's designed to *be used with them*, not *replace them*. But that doesn't mean it's not useful for other purposes. In the future, more configuration APIs will be added and `rathole` can be used with an external dashboard. > Make each program do one thing well. - *Domain based forwarding for HTTP* Introducing these kind of features into `rathole` itself ultimately reinvent a nginx. Use nginx to do this and set `rathole` as the upstream. This method achieves better performance as well as flexibility. - *HTTP Request Logging* `rathole` doesn't interference with the application layer traffic. A right place for this kind of stuff is the web server, and a network capture tool. - *`frp`'s STCP or other setup that requires visitors' side configuration* If that kind of setup is possible, then there are a lot more tools available. You may want to consider secure tunnels like wireguard or zerotier. `rathole` primarily focuses on NAT traversal by forwarding, which doesn't require any setup for visitors. - *Caching `local_ip`'s DNS records* As responded in [issue #183](https://github.com/rapiz1/rathole/issues/183), `local_ip` cache is not feasible because we have no reliable way to detect ip change. Handle DNS TTL and so on should be done with a DNS server, not a client. Caching ip is generally dangerous for clients. If you care about the `local_ip` query you can set up a local DNS server and enable caching. Then the local lookup should be trivial. --- ### Transport (`docs/transport.md`) # Security By default, `rathole` forwards traffic as it is. Different options can be enabled to secure the traffic. ## TLS Checkout the [example](../examples/tls) ### Client Normally, a self-signed certificate is used. In this case, the client needs to trust the CA. `trusted_root` is the path to the root CA's certificate PEM file. `hostname` is the hostname that the client used to validate aginst the certificate that the server presents. Note that it does not have to be the same with the `remote_addr` in `[client]`. ```toml [client.transport.tls] trusted_root = "example/tls/rootCA.crt" hostname = "localhost" ``` ### Server PKCS#12 archives are needed to run the server. It can be created using openssl like: ```sh openssl pkcs12 -export -out identity.pfx -inkey server.key -in server.crt -certfile ca_chain_certs.crt ``` Aruguments are: - `-inkey`: Server Private Key - `-in`: Server Certificate - `-certfile`: CA Certificate Creating self-signed certificate with one's own CA is a non-trival task. However, a script is provided under tls example folder for reference. ### Rustls Support `rathole` provides optional `rustls` support. [Build Guide](build-guide.md) demostrated this. One difference is that, the crate we use for loading PKCS#12 archives can only handle limited types of PBE algorithms. We only support PKCS#12 archives that they (crate `p12`) support. So we need to specify the legacy format (openssl 1.x format) when creating the PKCS#12 archive. In short, the command used with openssl 3 to create the PKCS#12 archive with `rustls` support is: ```sh openssl pkcs12 -export -out identity.pfx -inkey server.key -in server.crt -certfile ca_chain_certs.crt -legacy ``` ## Noise Protocol ### Quickstart for the Noise Protocl In one word, the [Noise Protocol](http://noiseprotocol.org/noise.html) is a lightweigt, easy to configure and drop-in replacement of TLS. No need to create a self-sign certificate to secure the connection. `rathole` comes with a reasonable default configuration for noise protocol. You can a glimpse of the minimal [example](../examples/noise_nk) for how it will look like. The default noise protocol that `rathole` uses, which is `Noise_NK_25519_ChaChaPoly_BLAKE2s`, providing the authentication of the server, just like TLS with properly configured certificates. So MITM is no more a problem. To use it, a X25519 keypair is needed. #### Generate a Keypair 1. Run `rathole --genkey`, which will generate a keypair using the default X25519 algorithm. It emits: ```sh $ rathole --genkey Private Key: cQ/vwIqNPJZmuM/OikglzBo/+jlYGrOt9i0k5h5vn1Q= Public Key: GQYTKSbWLBUSZiGfdWPSgek9yoOuaiwGD/GIX8Z1kkE= ``` (WARNING: Don't use the keypair from the Internet, including this one) 2. The server should keep the private key to identify itself. And the client should keep the public key, which is used to verify whether the peer is the authentic server. So relevant snippets of configuration are: ```toml # Client Side Configuration [client.transport] type = "noise" [client.transport.noise] remote_public_key = "GQYTKSbWLBUSZiGfdWPSgek9yoOuaiwGD/GIX8Z1kkE=" # Server Side Configuration [server.transport] type = "noise" [server.transport.noise] local_private_key = "cQ/vwIqNPJZmuM/OikglzBo/+jlYGrOt9i0k5h5vn1Q=" ``` Then `rathole` will run under the protection of the Noise Protocol. ## Specifying the Pattern of Noise Protocol The default configuration of Noise Protocol that comes with `rathole` satifies most use cases, which is described above. But there're other patterns that can be useful. ### No Authentication This configuration provides encryption of the traffic but provides no authentication, which means it's vulnerable to MITM attack, but is resistent to the sniffing and replay attack. If MITM attack is not one of the concerns, this is more convenient to use. ```toml # Server Side Configuration [server.transport.noise] pattern = "Noise_XX_25519_ChaChaPoly_BLAKE2s" # Client Side Configuration [client.transport.noise] pattern = "Noise_XX_25519_ChaChaPoly_BLAKE2s" ``` ### Bidirectional Authentication ```toml # Server Side Configuration [server.transport.noise] pattern = "Noise_KK_25519_ChaChaPoly_BLAKE2s" local_private_key = "server-priv-key-here" remote_public_key = "client-pub-key-here" # Client Side Configuration [client.transport.noise] pattern = "Noise_KK_25519_ChaChaPoly_BLAKE2s" local_private_key = "client-priv-key-here" remote_public_key = "server-pub-key-here" ``` ### Other Patterns To find out which pattern to use, refer to: - [7.5. Interactive handshake patterns (fundamental)](https://noiseprotocol.org/noise.html#interactive-handshake-patterns-fundamental) - [8. Protocol names and modifiers](https://noiseprotocol.org/noise.html#protocol-names-and-modifiers) Note that PSKs are not supported currently. Free to open an issue if you need it. --- --- METRICS --- - Files Extracted: 7 - Estimated Token Budget: ~5658 tokens - Canonical Reference: https://codewiki.google/github.com/rathole-org/rathole