serf Documentation

Service orchestration and management tool.

6,075 stars Go Go 6 files · ~3,572 tokens
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy serf on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name serf --restart always hashicorp/serf:latest

Technical Documentation: hashicorp/serf

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

Serf [](https://github.com/hashicorp/serf/actions) [](https://gitter.im/hashicorp-serf/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

[!NOTE]
Looking for serf.io? The Serf website was shut down on 10/02/2024. The docs
previously served from serf.io can be found
https://github.com/hashicorp/serf/blob/master/docs/index.html.markdown

Serf is a decentralized solution for service discovery and orchestration
that is lightweight, highly available, and fault tolerant.

Serf runs on Linux, Mac OS X, and Windows. An efficient and lightweight gossip
protocol is used to communicate with other nodes. Serf can detect node failures
and notify the rest of the cluster. An event system is built on top of
Serf, letting you use Serf's gossip protocol to propagate events such
as deploys, configuration changes, etc. Serf is completely masterless
with no single point of failure.

Here are some example use cases of Serf, though there are many others:

  • Discovering web servers and automatically adding them to a load balancer
  • Organizing many memcached or redis nodes into a cluster, perhaps with
    something like twemproxy or
    maybe just configuring an application with the address of all the
    nodes
  • Triggering web deploys using the event system built on top of Serf
  • Propagating changes to configuration to relevant nodes.
  • Updating DNS records to reflect cluster changes as they occur.
  • Much, much more.

Quick Start

First, download a pre-built Serf binary
for your operating system, compile Serf yourself, or install
using go get -u github.com/hashicorp/serf/cmd/serf.

Next, let's start a couple Serf agents. Agents run until they're told to quit
and handle the communication of maintenance tasks of Serf. In a real Serf
setup, each node in your system will run one or more Serf agents (it can
run multiple agents if you're running multiple cluster types. e.g. web
servers vs. memcached servers).

Start each Serf agent in a separate terminal session so that we can see
the output of each. Start the first agent:

text
$ serf agent -node=foo -bind=127.0.0.1:5000 -rpc-addr=127.0.0.1:7373
...

Start the second agent in another terminal session (while the first is still
running):

text
$ serf agent -node=bar -bind=127.0.0.1:5001 -rpc-addr=127.0.0.1:7374
...

At this point two Serf agents are running independently but are still
unaware of each other. Let's now tell the first agent to join an existing
cluster (the second agent). When starting a Serf agent, you must join an
existing cluster by specifying at least one existing member. After this,
Serf gossips and the remainder of the cluster becomes aware of the join.
Run the following commands in a third terminal session.

text
$ serf join 127.0.0.1:5001
...

If you're watching your terminals, you should see both Serf agents
become aware of the join. You can prove it by running serf members
to see the members of the Serf cluster:

text
$ serf members
foo    127.0.0.1:5000    alive
bar    127.0.0.1:5001    alive
...

At this point, you can ctrl-C or force kill either Serf agent, and they'll
update their membership lists appropriately. If you ctrl-C a Serf agent,
it will gracefully leave by notifying the cluster of its intent to leave.
If you force kill an agent, it will eventually (usually within seconds)
be detected by another member of the cluster which will notify the
cluster of the node failure.

Documentation

Full, comprehensive documentation is viewable on the Serf website:

https://github.com/hashicorp/serf/tree/master/docs

Developing Serf

If you wish to work on Serf itself, you'll first need Go
installed (stable version is required). Make sure you have Go properly
installed,
including setting up your GOPATH.

Next, clone this repository into $GOPATH/src/github.com/hashicorp/serf and
then just type make. In a few moments, you'll have a working serf executable:

text
$ make
...
$ bin/serf
...

NOTE: make will also place a copy of the executable under $GOPATH/bin/

Serf is first and foremost a library with a command-line interface, serf. The
Serf library is independent of the command line agent, serf. The serf
binary is located under cmd/serf and can be installed stand alone by issuing
the command go get -u github.com/hashicorp/serf/cmd/serf. Applications using
the Serf library should only need to include github.com/hashicorp/serf.

Tests can be run by typing make test.

If you make any changes to the code, run make format in order to automatically
format the code according to Go standards.

Metrics Emission and Compatibility

This library can emit metrics using either github.com/armon/go-metrics or github.com/hashicorp/go-metrics. Choosing between the libraries is controlled via build tags.

Build Tags

  • armonmetrics - Using this tag will cause metrics to be routed to armon/go-metrics
  • hashicorpmetrics - Using this tag will cause all metrics to be routed to hashicorp/go-metrics

If no build tag is specified, the default behavior is to use armon/go-metrics.

Deprecating armon/go-metrics

Emitting metrics to armon/go-metrics is officially deprecated. Usage of armon/go-metrics will remain the default until mid-2025 with opt-in support continuing to the end of 2025.

Migration
To migrate an application currently using the older armon/go-metrics to instead use hashicorp/go-metrics the following should be done.

  1. Upgrade libraries using armon/go-metrics to consume hashicorp/go-metrics/compat instead. This should involve only changing import statements. All repositories in the hashicorp namespace
  2. Update an applications library dependencies to those that have the compatibility layer configured.
  3. Update the application to use hashicorp/go-metrics for configuring metrics export instead of armon/go-metrics
    • Replace all application imports of github.com/armon/go-metrics with github.com/hashicorp/go-metrics
    • Instrument your build system to build with the hashicorpmetrics tag.

Eventually once the default behavior changes to use hashicorp/go-metrics by default (mid-2025), you can drop the hashicorpmetrics build tag.

Overview (`client/README.md`)

Serf Client

This repo provide the client package, which is used to interact with
a Serf agent using the msgpack RPC system it supports. This is the official
reference implementation, and is used inside the Serf CLI to support the various
commands.

Full documentation can be found on godoc here.

Overview (`dist/README.md`)

dist

This folder is used to share files that are not part of the Serf binary, but are
useful for operational purposes. For example, systemd service unit files.

Debian/Ubuntu package metadata

Move the debian directory to the root of the repo, and run dpkg-buildpackage.

Overview (`demo/vagrant-cluster/README.md`)

Vagrant Serf Demo

This demo provides a very simple Vagrantfile that creates two nodes,
one at "172.20.20.10" and another at "172.20.20.11". Both are running
a standard Ubuntu 12.04 distribution, with Serf pre-installed under
/usr/bin/serf.

To get started, you can start the cluster by just doing:

text
$ vagrant up

Once it is finished, you should be able to see the following:

text
$ vagrant status
Current machine states:
n1                        running (vmware_fusion)
n2                        running (vmware_fusion)

At this point the two nodes are running and you can SSH in to play with them:

text
$ vagrant ssh n1
...
$ vagrant ssh n2
...

To learn more about starting serf, joining nodes and interacting with the agent,
checkout the getting started guide.

Overview (`demo/web-load-balancer/README.md`)

Serf Demo: Web Servers + Load Balancer

In this demo, Serf is used to coordinate web servers entering a load
balancer. The nodes in this example can come up in any order and they'll
eventually converge to a running cluster. When visiting the load balancer,
you'll see each of the nodes visited in round-robin.

After telling the cluster to launch, no further human interaction is
required. The cluster will launch and orchestrate itself into a load
balanced web server setup.

The nodes are launched on AWS using CloudFormation and therefore will
incur a real cost. You can control this cost based on how many nodes
you decide to launch.

Running the Demo

To run the demo, log into your AWS console, go to the CloudFormation
tab, and create a new stack using the "cloudformation.json" file in this
directory.

Configure how many web nodes you'd like to launch then create the stack.
When the stack is finished creating, the load balancer IP will be in
the "outputs" tab of that stack.

Visit http://IP:9999/ to see the HAProxy stats page and list of
configured backends. Or visit the IP directly in your browser to see
it round robin between the web nodes.

It will take some time for the load balancer and web servers to install
their software and configure themselves. If the load balancer stats page
is not up yet, give it a few seconds and try again. Once it is up, it
will automatically refresh so you can watch as nodes eventually come
online.

How it Works

Node Setup

All the nodes (webs and load balancer) come up at the same time, with no
explicit ordering defined. If there is an order, it doesn't actually matter
for the function of the demo. The web servers can all come up before the load
balancer, the load balancer can come up in the middle of some web servers, etc.

Each of the nodes does an initial setup using a shell script. In a real
environment, this initial setup would probably be done with a real
configuration management system. For the purposes of this demo, we use
shell scripts for ease of understanding.

The load balancer runs setup_load_balancer.sh which installs HAProxy.
HAProxy is configured with no backend servers. Serf will handle this later.

The web server runs setup_web_server.sh which installs Apache and
sets up an index.html to just say what host it is.

At this stage, the load balancer is running HAProxy, and the web servers
are running Apache, but they're not aware of each other in any way.

Next, all nodes run setup_serf.sh which installs Serf and configures it
to run. This script also starts the Serf agent. For web servers, it also
starts a task that continuosly tries to join the Serf cluster at address
10.0.0.5, which happens to be the static IP assigned to the load balancer.
Note that Serf can join any node in the cluster. We just set a static IP
on the load balancer for ease of automation, but in a real production
environment you can use any existing node, potentially just using DNS.

Web server roles also make use of a "serf-query" upstart task to dynamically
update the index page they serve. By using a "load" query, each node asks
the rest of the cluster for their current load. This information is used
to dynamically generate the index page.

Serf Configuration

The Serf configuration is very simple. Each node runs a Serf agent. The
web servers run the agent with a role of "web" and the load balancer runs
the agent with a role of "lb".

Event: member-join

On member-join, the following shell script is run. We'll talk about the
function of the shell script after the code sample.

sh
if [ "x${SERF_TAG_ROLE}" != "xlb" ]; then
    echo "Not an lb. Ignoring member join."
    exit 0
fi

while read line; do
    ROLE=`echo $line | awk '{print \$3 }'`
    if [ "x${ROLE}" != "xweb" ]; then
        continue
    fi

    echo $line | \
        awk '{ printf "    server %s %s check\n", $1, $2 }' >>/etc/haproxy/haproxy.cfg
done

/etc/init.d/haproxy reload

The script first checks if this node is a load balancer. If it isn't a load
balancer, we don't do anything on the "member-join" event. Web servers simply
don't care about other members.

Otherwise, if this is a load balancer, then it handles the "member-join" event
and reads the event data and updates the configuration of HAProxy. In a
production environment, you might use something like a template processor to do
this rather than appending to the configuration. For the purposes of this demo,
this works as well.

Finally, HAProxy is reloaded so the configuration is read.

The result of this is that as members join, they're automatically added into
rotation on the load balancer.

Event: member-leave, member-failed

On member-leave and member-failed evets, the following shell script is run.
This demo doesn't differentiate between the two events, treating both
the same way and simply removing the node from the HAProxy configuration.

sh
if [ "x${SERF_TAG_ROLE}" != "xlb" ]; then
    echo "Not an lb. Ignoring member leave"
    exit 0
fi

while read line; do
    NAME=`echo $line | awk '{print \$1 }'`
    sed -i'' "/${NAME} /d" /etc/haproxy/haproxy.cfg
done

/etc/init.d/haproxy reload

This script also does nothing if this node is not a load balanacer. Otherwise,
it uses sed to remove each of the nodes from the HAProxy configuration and
reloads it.

Overview (`demo/web-load-balancer/user_data/README.md`)

This directory contains the user data that is used to initialize the
nodes that are started by CloudFormation.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →