rainbond Documentation

Technical Documentation: goodrain/rainbond

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

What is Rainbond?

Rainbond is an open-source container platform that requires no Kubernetes expertise. Its core capabilities are 100% open source.

It abstracts away infrastructure complexity and provides a unified way to deploy and manage business applications, AI-generated projects, open-source AI software, and large language model services. AI helps teams deploy and operate these workloads, keeping applications running reliably on their own servers or Kubernetes clusters.

Where to start

Your goal	Start here
I use AI coding and want to deploy my project	Install RainSkills
I want to try Rainbond without preparing a server	Use Rainbond Cloud
I want to run applications on my own servers or Kubernetes	Install Rainbond privately
I want to deploy open-source applications such as Dify or RAGFlow	Visit the Rainbond Application Marketplace
I am evaluating open-source container platforms	Explore Rainbond's application management and delivery capabilities

What problems Rainbond solves

1. Deliver applications without deeply learning Kubernetes

Rainbond supports deploying applications from source code, container images, Docker Compose, Helm, or application templates. Its graphical interface and standardized workflows bring application dependencies, access, upgrades, and rollbacks into one place.

2. Make delivery in complex enterprise environments more reliable

Rainbond is especially suitable for:

Private deployment
Internal-network deployment
Offline delivery
Xinchuang adaptation
x86 to ARM migration validation

3. Standardize delivery with marketplace and templates

Rainbond is not only about “getting apps running”. It also provides:

Application templates
Application marketplace
One-click installation
One-click upgrade
Customer environment replication

4. Keep operations and delivery centered around the application

Rainbond emphasizes:

Application-level abstraction
Application topology and dependencies
Application upgrades and rollbacks
Delivery and operation across multiple environments and clusters

Why Rainbond

Lower barrier, not lower capability

Rainbond does not simply hide Kubernetes. It redistributes complexity:

The platform and operations teams absorb low-level concerns
Development and delivery teams focus on building, deploying, delivering, and operating applications

Better fit for complex enterprise environments

Many platforms are strong at cluster management.
Rainbond is stronger when the real problem is:

How to deliver applications
How to replicate delivery to customer environments
How to handle upgrades in offline environments
How to privately deploy AI applications

Stronger marketplace and standardized delivery capability

If what you really need is:

Template-based delivery
Marketplace reuse
Offline package export/import
Versioned upgrades and rollbacks

Rainbond is closer to the actual work.

How it differs from common platform choices

Dimension	Rainbond	Rancher / KubeSphere type platforms
Primary focus	Application delivery, application management, template reuse	Cluster governance, platform ops, Kubernetes management
Typical users	Developers, delivery teams, enterprise IT, platform owners	K8s admins, platform ops, cluster governance teams
Strongest scenarios	Private deployment, offline, Xinchuang, marketplace delivery, AI privatization	Multi-cluster governance, resource management, platform ops
Learning curve	Lower, exposes less K8s detail	Requires stronger K8s and platform governance knowledge

If you are in evaluation mode, start here:

Typical scenarios

Deliver applications without deeply learning Kubernetes
Privately deploy AI applications
Offline and internal-network delivery
Xinchuang and ARM migration
Enterprise marketplace and standardized delivery

Continue with:

Quick start

Requirements

Linux or macOS
Recommended: at least 2 CPU / 8GB RAM / 50GB disk

Quick install

Run:

bash

curl -o install.sh https://get.rainbond.com && bash ./install.sh

Then open:

bash

http://<your-ip>:7070

Next step

Community and support

Contributing

If you want to contribute, start here:

We welcome:

Code contributions
Documentation improvements
User stories and technical writeups
Application template and plugin sharing

rainbond-console - Console backend
rainbond-ui - Console frontend
rainbond-operator - Installation and operations
builder - Source build toolset

License

This repository is licensed under the Rainbond Open Source License, based on Apache 2.0 with additional conditions.

Overview (`LICENSES/README.md`)

Third-Party Licenses

This directory contains license files for third-party components used in this project.

Directory Structure

apache/ - Apache Software Foundation licensed components
apecloud/ - ApeCloud licensed components
minio/ - MinIO licensed components
rancher/ - Rancher licensed components

Purpose

These licenses are maintained to ensure compliance with the licensing requirements of all third-party software integrated into this project.

Overview (`plugins/README.md`)

Rainbond Plugins

This directory contains various plugin components of the Rainbond platform, which are used to expand and enhance the functionality of Rainbond.

License

This project follows the same license as the Rainbond main project.

Overview (`third_party/coreos-etcd/README.md`)

etcd

Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order to get stable binaries.

the etcd v2 documentation has moved

etcd is a distributed reliable key-value store for the most critical data of a distributed system, with a focus on being:

Simple: well-defined, user-facing API (gRPC)
Secure: automatic TLS with optional client cert authentication
Fast: benchmarked 10,000 writes/sec
Reliable: properly distributed using Raft

etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.

etcd is used in production by many companies, and the development team stands behind it in critical deployment scenarios, where etcd is frequently teamed with applications such as Kubernetes, fleet, locksmith, vulcand, Doorman, and many others. Reliability is further ensured by rigorous testing.

See etcdctl for a simple command line client.

Community meetings

etcd contributors and maintainers have bi-weekly meetings at 11:00 AM (USA Pacific) on Tuesdays. There is an iCalendar format for the meetings here. Anyone is welcome to join via Zoom or audio-only: +1 669 900 6833. An initial agenda will be posted to the shared Google docs a day before each meeting, and everyone is welcome to suggest additional topics or other agendas.

Getting started

Getting etcd

The easiest way to get etcd is to use one of the pre-built release binaries which are available for OSX, Linux, Windows, rkt, and Docker. Instructions for using these binaries are on the GitHub releases page.

For those wanting to try the very latest version, build the latest version of etcd from the master branch. This first needs Go installed (version 1.9+ is required). All development occurs on master, including new features and bug fixes. Bug fixes are first targeted at master and subsequently ported to release branches, as described in the branch management guide.

Running etcd

First start a single-member cluster of etcd.

If etcd is installed using the pre-built release binaries, run it from the installation location as below:

/tmp/etcd-download-test/etcd

The etcd command can be simply run as such if it is moved to the system path as below:

mv /tmp/etcd-download-test/etcd /usr/locale/bin/

etcd

If etcd is build from the master branch, run it as below:

./bin/etcd

This will bring up etcd listening on port 2379 for client communication and on port 2380 for server-to-server communication.

Next, let's set a single key, and then retrieve it:

text

ETCDCTL_API=3 etcdctl put mykey "this is awesome"
ETCDCTL_API=3 etcdctl get mykey

That's it! etcd is now running and serving client requests. For more

etcd TCP ports

The official etcd ports are 2379 for client requests, and 2380 for peer communication.

Running a local etcd cluster

First install goreman, which manages Procfile-based applications.

Our Procfile script will set up a local example cluster. Start it with:

goreman start

This will bring up 3 etcd members infra1, infra2 and infra3 and etcd grpc-proxy, which runs locally and composes a cluster.

Every cluster member and proxy accepts key value reads and key value writes.

Running etcd on Kubernetes

To run an etcd cluster on Kubernetes, try etcd operator.

Next steps

Now it's time to dig into the full etcd API and other guides.

Read the full documentation.
Explore the full gRPC API.
Set up a multi-machine cluster.
Learn the config format, env variables and flags.
Find language bindings and tools.
Use TLS to secure an etcd cluster.
Tune etcd.

Contact

Mailing list: etcd-dev
IRC: #etcd on freenode.org
Planning/Roadmap: milestones, roadmap
Bugs: issues

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

Reporting bugs

See reporting bugs for details about reporting any issues.

License

etcd is under the Apache 2.0 license. See the LICENSE file for details.

Overview (`third_party/process-exporter/README.md`)

process-exporter

Prometheus exporter that mines /proc to report on selected processes.

Some apps are impractical to instrument directly, either because you
don't control the code or they're written in a language that isn't easy to
instrument with Prometheus. We must instead resort to mining /proc.

Installation

Either grab a package for your OS from the Releases page, or
install via docker.

Running

Usage:

text

process-exporter [options] -config.path filename.yml

or via docker:

text

docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`:/config ncabatoff/process-exporter --procfs /host/proc -config.path /config/filename.yml

Important options (run process-exporter --help for full list):

-children (default:true) makes it so that any process that otherwise
isn't part of its own group becomes part of the first group found (if any) when
walking the process tree upwards. In other words, resource usage of
subprocesses is added to their parent's usage unless the subprocess identifies
as a different group name.

-threads (default:true) means that metrics will be broken down by thread name
as well as group name.

-recheck (default:false) means that on each scrape the process names are
re-evaluated. This is disabled by default as an optimization, but since
processes can choose to change their names, this may result in a process
falling into the wrong group if we happen to see it for the first time before
it's assumed its proper name.

-procnames is intended as a quick alternative to using a config file. Details
in the following section.

To disable any of these options, use the -option=false.

Configuration and group naming

To select and group the processes to monitor, either provide command-line
arguments or use a YAML configuration file.

The recommended option is to use a config file via -config.path, but for
convenience and backwards compatibility the -procnames/-namemapping options
exist as an alternative.

Using a config file

The general format of the -config.path YAML file is a top-level
process_names section, containing a list of name matchers:

text

process_names:
  - matcher1
  - matcher2
  ...
  - matcherN

The default config shipped with the deb/rpm packages is:

text

process_names:
  - name: "{{.Comm}}"
    cmdline:
    - '.+'

A process may only belong to one group: even if multiple items would match, the
first one listed in the file wins.

(Side note: to avoid confusion with the cmdline YAML element, we'll refer to
the command-line arguments of a process /proc//cmdline as the array
argv[].)

Using a config file: group name

Each item in process_names gives a recipe for identifying and naming
processes. The optional name tag defines a template to use to name
matching processes; if not specified, name defaults to {{.ExeBase}}.

Template variables available:

{{.Comm}} contains the basename of the original executable, i.e. 2nd field in /proc//stat
{{.ExeBase}} contains the basename of the executable
{{.ExeFull}} contains the fully qualified path of the executable
{{.Username}} contains the username of the effective user
{{.Matches}} map contains all the matches resulting from applying cmdline regexps
{{.PID}} contains the PID of the process. Note that using PID means the group
will only contain a single process.
{{.StartTime}} contains the start time of the process. This can be useful
in conjunction with PID because PIDs get reused over time.

Using PID or StartTime is discouraged: this is almost never what you want,
and is likely to result in high cardinality metrics which Prometheus will have
trouble with.

Using a config file: process selectors

Each item in process_names must contain one or more selectors (comm, exe
or cmdline); if more than one selector is present, they must all match. Each
selector is a list of strings to match against a process's comm, argv[0],
or in the case of cmdline, a regexp to apply to the command line. The cmdline
regexp uses the Go syntax.

For comm and exe, the list of strings is an OR, meaning any process
matching any of the strings will be added to the item's group.

For cmdline, the list of regexes is an AND, meaning they all must match. Any
capturing groups in a regexp must use the ?P<name> option to assign a name to
the capture, which is used to populate .Matches.

Performance tip: give an exe or comm clause in addition to any cmdline
clause, so you avoid executing the regexp when the executable name doesn't
match.

text

process_names:
  # comm is the second field of /proc/<pid>/stat minus parens.
  # It is the base executable name, truncated at 15 chars.
  # It cannot be modified by the program, unlike exe.
  - comm:
    - bash

  # exe is argv[0]. If no slashes, only basename of argv[0] need match.
  # If exe contains slashes, argv[0] must match exactly.
  - exe:
    - postgres
    - /usr/local/bin/prometheus

  # cmdline is a list of regexps applied to argv.
  # Each must match, and any captures are added to the .Matches map.
  - name: "{{.ExeFull}}:{{.Matches.Cfgfile}}"
    exe:
    - /usr/local/bin/process-exporter
    cmdline:
    - -config.path\s+(?P<Cfgfile>\S+)

Here's the config I use on my home machine:

text

process_names:
  - comm:
    - chromium-browse
    - bash
    - prometheus
    - gvim
  - exe:
    - /sbin/upstart
    cmdline:
    - --user
    name: upstart:-user

Using -procnames/-namemapping instead of config.path

Every name in the procnames list becomes a process group. The default name of
a process is the value found in the second field of /proc//stat
("comm"), which is truncated at 15 chars. Usually this is the same as the
name of the executable.

If -namemapping isn't provided, every process with a comm value present
in -procnames is assigned to a group based on that name, and any other
processes are ignored.

The -namemapping option is a comma-separated list of alternating
name,regexp values. It allows assigning a name to a process based on a
combination of the process name and command line. For example, using

-namemapping "python2,([^/]+).py,java,-jar\s+([^/]+).jar"

will make it so that each different python2 and java -jar invocation will be
tracked with distinct metrics. Processes whose remapped name is absent from
the procnames list will be ignored. On a Ubuntu Xenian machine being used as
a workstation, here's a good way of tracking resource usage for a few
different key user apps:

process-exporter -namemapping "upstart,(--user)"
-procnames chromium-browse,bash,gvim,prometheus,process-exporter,upstart:-user

Since upstart --user is the parent process of the X11 session, this will
make all apps started by the user fall into the group named "upstart:-user",
unless they're one of the others named explicitly with -procnames, like gvim.

Group Metrics

There's no meaningful way to name a process that will only ever name a single process, so process-exporter assumes that every metric will be attached
to a group of processes - not a
process group in the technical
sense, just one or more processes that meet a configuration's specification
of what should be monitored and how to name it.

All these metrics start with namedprocess_namegroup_ and have at minimum
the label groupname.

num_procs gauge

Number of processes in this group.

cpu_seconds_total counter

CPU usage based on /proc/[pid]/stat fields utime(14) and stime(15) i.e. user and system time. This is similar to the node_exporter's node_cpu_seconds_total.

read_bytes_total counter

Bytes read based on /proc/[pid]/io field read_bytes. The man page
says

Attempt to count the number of bytes which this process really did cause to be fetched from the storage layer. This is accurate for block-backed filesystems.

but I would take it with a grain of salt.

write_bytes_total counter

Bytes written based on /proc/[pid]/io field write_bytes. As with
read_bytes, somewhat dubious. May be useful for isolating which processes
are doing the most I/O, but probably not measuring just how much I/O is happening.

major_page_faults_total counter

Number of major page faults based on /proc/[pid]/stat field majflt(12).

minor_page_faults_total counter

Number of minor page faults based on /proc/[pid]/stat field minflt(10).

context_switches_total counter

Number of context switches based on /proc/[pid]/status fields voluntary_ctxt_switches
and nonvoluntary_ctxt_switches. The extra label ctxswitchtype can have two values:
voluntary and nonvoluntary.

memory_bytes gauge

Number of bytes of memory used. The extra label memtype can have three values:

resident: Field rss(24) from /proc/[pid]/stat, whose doc says:

This is just the pages which count toward text, data, or stack space. This does not include pages which have not been demand-loaded in, or which are swapped out.

virtual: Field vsize(23) from /proc/[pid]/stat, virtual memory size.

swapped: Field VmSwap from /proc/[pid]/status, translated from KB to bytes.

If gathering smaps file is enabled, two additional values for memtype are added:

proportionalResident: Sum of "Pss" fields from /proc/[pid]/smaps, whose doc says:

The "proportional set size" (PSS) of a process is the count of pages it has
in memory, where each page is divided by the number of processes sharing it.

proportionalSwapped: Sum of "SwapPss" fields from /proc/[pid]/smaps

open_filedesc gauge

Number of file descriptors, based on counting how many entries are in the directory
/proc/[pid]/fd.

worst_fd_ratio gauge

Worst ratio of open filedescs to filedesc limit, amongst all the procs in the
group. The limit is the fd soft limit based on /proc/[pid]/limits.

Normally Prometheus metrics ought to be as "basic" as possible (i.e. the raw
values rather than a derived ratio), but we use a ratio here because nothing
else makes sense. Suppose there are 10 procs in a given group, each with a
soft limit of 4096, and one of them has 4000 open fds and the others all have
40, their total fdcount is 4360 and total soft limit is 40960, so the ratio
is 1:10, but in fact one of the procs is about to run out of fds. With
worst_fd_ratio we're able to know this: in the above example it would be
0.97, rather than the 0.10 you'd see if you computed sum(open_filedesc) /
sum(limit_filedesc).

oldest_start_time_seconds gauge

Epoch time (seconds since 1970/1/1) at which the oldest process in the group
started. This is derived from field starttime(22) from /proc/[pid]/stat, added
to boot time to make it relative to epoch.

num_threads gauge

Sum of number of threads of all process in the group. Based on field num_threads(20)
from /proc/[pid]/stat.

states gauge

Number of threads in the group in each of various states, based on the field
state(3) from /proc/[pid]/stat.

The extra label state can have these values: Running, Sleeping, Waiting, Zombie, Other.

Group Thread Metrics

Since publishing thread metrics adds a lot of overhead, use the -threads command-line argument to disable them,
if necessary.

All these metrics start with namedprocess_namegroup_ and have at minimum
the labels groupname and threadname. threadname is field comm(2) from
/proc/[pid]/stat. Just as groupname breaks the set of processes down into
groups, threadname breaks a given process group down into subgroups.

thread_count gauge

Number of threads in this thread subgroup.

thread_cpu_seconds_total counter

Same as cpu_user_seconds_total and cpu_system_seconds_total, but broken down
per-thread subgroup. Unlike cpu_user_seconds_total/cpu_system_seconds_total,
the label cpumode is used to distinguish between user and system time.

thread_io_bytes_total counter

Same as read_bytes_total and write_bytes_total, but broken down
per-thread subgroup. Unlike read_bytes_total/write_bytes_total,
the label iomode is used to distinguish between read and write bytes.

thread_major_page_faults_total counter

Same as major_page_faults_total, but broken down per-thread subgroup.

thread_minor_page_faults_total counter

Same as minor_page_faults_total, but broken down per-thread subgroup.

thread_context_switches_total counter

Same as context_switches_total, but broken down per-thread subgroup.

Instrumentation cost

process-exporter will consume CPU in proportion to the number of processes in
the system and the rate at which new ones are created. The most expensive
parts - applying regexps and executing templates - are only applied once per
process seen, unless the command-line option -recheck is provided.

If you have mostly long-running processes process-exporter overhead should be
minimal: each time a scrape occurs, it will parse of /proc/$pid/stat and
/proc/$pid/cmdline for every process being monitored and add a few numbers.

Dashboards

An example Grafana dashboard to view the metrics is available at https://grafana.net/dashboards/249

Building

Requires Go 1.12 installed.

text

make

Overview (`util/http/README.md`)

用法实例

text

type User struct {
	Username string `validate:"username|required|between:3,8"`
	Email    string `validate:"email|email"`
	Web      string `validate:"web|url"`
	Age      int    `validate:"age|numeric_between:18,55"`
}

func handler(w http.ResponseWriter, r *http.Request) {
	var user User
	messages := validator.MapData{
		"username": []string{"required:You must provide username", "between:username must be between 3 to 8 chars"},
		"web":      []string{"url:You must provide a valid url"},
	}

	if !httputil.ValidatorRequestStructAndErrorResponse(r, w, &user, messages) {
		return
	}
	httputil.ReturnSuccess(r, w, user)
}
func handlerMap(w http.ResponseWriter, r *http.Request) {
	rule := validator.MapData{
		"username": []string{"required", "between:3,5"},
		"web":      []string{"url"},
	}
	messages := validator.MapData{
		"username": []string{"required:You must provide username", "between:username must be between 3 to 8 chars"},
		"web":      []string{"url:You must provide a valid url"},
	}
	data, ok := httputil.ValidatorRequestMapAndErrorResponse(r, w, rule, messages)
	if !ok {
		return
	}
	httputil.ReturnSuccess(r, w, data)
}

Validation Rules

alpha The field under validation must be entirely alphabetic characters.
alpha_dash The field under validation may have alpha-numeric characters, as well as dashes and underscores.
alpha_num The field under validation must be entirely alpha-numeric characters.
numeric The field under validation must be entirely numeric characters.
numeric_between:int,int The field under validation must be a numeric value between the range.
e.g: numeric_between:18,65 may contains numeric value like 35, 55
bool The field under validation must be able to be cast as a boolean. Accepted input are true, false, 1, 0, "1" and "0".
credit_card The field under validation must have a valid credit card number. Accepted cards are Visa, MasterCard, American Express, Diners Club, Discover and JCB card
coordinate The field under validation must have a value of valid coordinate.
css_color The field under validation must have a value of valid CSS color. Accepted colors are hex, rgb, rgba, hsl, hsla like #909, #00aaff, rgb(255,122,122)
date The field under validation must have a valid date of format yyyy-mm-dd or yyyy/mm/dd.
date:dd-mm-yyyy The field under validation must have a valid date of format dd-mm-yyyy.
digits:int The field under validation must be numeric and must have an exact length of value.
digits_between:int,int The field under validation must be numeric and must have length between the range.
e.g: digits_between:3,5 may contains digits like 2323, 12435
email The field under validation must have a valid email.
float The field under validation must have a valid float number.
in:foo,bar The field under validation must have one of the values. e.g: in:admin,manager,user must contain the values (admin or manager or user)
min:int The field under validation must have a min length of characters.
e.g: min:3 may contains characters minimum length of 3 like "john", "jane", "jane321" but not "mr", "xy"
max:int The field under validation must have a max length of characters.
e.g: max:6 may contains characters maximum length of 6 like "john doe", "jane doe" but not "john", "jane"
not_in:foo,bar The field under validation must have one value except foo,bar. e.g: not_in:admin,manager,user must not contain the values (admin or manager or user)
len:int The field under validation must have an exact length of characters.
e.g: len:4 may contains characters exact length of 4 like Food, Mood, Good
ip The field under validation must be a valid IP address.
ip_v4 The field under validation must be a valid IP V4 address.
ip_v6 The field under validation must be a valid IP V6 address.
json The field under validation must be a valid JSON string.
lat The field under validation must be a valid latitude.
lon The field under validation must be a valid longitude.
regex:regurlar expression The field under validation validate against the regex. e.g: regex:^[a-zA-Z]+$ validate the letters.
required The field under validation must be present in the input data and not empty. A field is considered "empty" if one of the following conditions are true: 1) The value is null. 2)The value is an empty string.
url The field under validation must be a valid URL.
uuid The field under validation must be a valid UUID.
uuid_v3 The field under validation must be a valid UUID V3.
uuid_v4 The field under validation must be a valid UUID V4.
uuid_v5 The field under validation must be a valid UUID V5.

Overview (`util/termtables/README.md`)

Termtables

A Go port of the Ruby library terminal-tables for
fast and simple ASCII table generation.

Installation

bash

go get github.com/apcera/termtables

Go Style Documentation

http://godoc.org/github.com/apcera/termtables

APC Command Line usage

--markdown — output a markdown table, e.g. apc app list --markdown

--html — output an html table, e.g. apc app list --html

--ascii — output an ascii table, e.g. apc app list --ascii

Basic Usage

package main

import (
  "fmt"
  "github.com/apcera/termtables"
)

func main() {
  table := termtables.CreateTable()

  table.AddHeaders("Name", "Age")
  table.AddRow("John", "30")
  table.AddRow("Sam", 18)
  table.AddRow("Julie", 20.14)

  fmt.Println(table.Render())
}

Result:

text

+-------+-------+
| Name  | Age   |
+-------+-------+
| John  | 30    |
| Sam   | 18    |
| Julie | 20.14 |
+-------+-------+

Advanced Usage

The package function-call EnableUTF8() will cause any tables created after
that point to use Unicode box-drawing characters for the table lines.

Calling EnableUTF8PerLocale() uses the C library's locale functionality to
determine if the current locale environment variables say that the current
character map is UTF-8. If, and only if, so, then EnableUTF8() will be
called.

Calling SetModeHTML(true) will cause any tables created after that point
to be emitted in HTML, while SetModeMarkdown(true) will trigger Markdown.
Neither should result in changes to later API to get the different results;
the primary intended use-case is extracting the same table, but for
documentation.

The table method .AddSeparator() inserts a rule line in the output. This
only applies in normal terminal output mode.

The table method .AddTitle() adds a title to the table; in terminal output,
this is an initial row; in HTML, it's a caption. In Markdown, it's a line of
text before the table, prefixed by Table: .

The table method .SetAlign() takes an alignment and a column number
(indexing starts at 1) and changes all current cells in that column to have
the given alignment. It does not change the alignment of cells added to the
table after this call. Alignment is only stored on a per-cell basis.

Known Issues

Normal output:

.SetAlign() does not affect headers.

Markdown output mode:

When emitting Markdown, the column markers are not re-flowed if a vertical
bar is an element of a cell, causing an escape to take place; since Markdown
is often converted to HTML, this only affects text viewing.
A title in Markdown is not escaped against all possible forms of Markdown
markup (to avoid adding a dependency upon a Markdown library, as supported
syntax can vary).
Markdown requires headers, so a dummy header will be inserted if needed.
Table alignment is not reflected in Markdown output.

Overview (`hack/k8s/codegen/README.md`)

How to use codegen

./hack/k8s/codegen/update-generated.sh

It should print:

text

Generating deepcopy funcs
Generating clientset for etcd:v1beta2 at github.com/coreos/etcd-operator/pkg/generated/clientset
Generating listers for etcd:v1beta2 at github.com/coreos/etcd-operator/pkg/generated/listers
Generating informers for etcd:v1beta2 at github.com/coreos/etcd-operator/pkg/generated/informers

Overview (`third_party/coreos-etcd/client/README.md`)

etcd/client

etcd/client is the Go client library for etcd.

etcd uses cmd/vendor directory to store external dependencies, which are
to be compiled into etcd release binaries. client can be imported without
vendoring. For full compatibility, it is recommended to vendor builds using
etcd's vendored packages, using tools like godep, as in
vendor directories.
For more detail, please read Go vendor design.

Install

bash

go get github.com/coreos/etcd/client

Usage

package main

import (
	"log"
	"time"
	"context"

	"github.com/coreos/etcd/client"
)

func main() {
	cfg := client.Config{
		Endpoints:               []string{"http://127.0.0.1:2379"},
		Transport:               client.DefaultTransport,
		// set timeout per request to fail fast when the target endpoint is unavailable
		HeaderTimeoutPerRequest: time.Second,
	}
	c, err := client.New(cfg)
	if err != nil {
		log.Fatal(err)
	}
	kapi := client.NewKeysAPI(c)
	// set "/foo" key with "bar" value
	log.Print("Setting '/foo' key with 'bar' value")
	resp, err := kapi.Set(context.Background(), "/foo", "bar", nil)
	if err != nil {
		log.Fatal(err)
	} else {
		// print common key info
		log.Printf("Set is done. Metadata is %q\n", resp)
	}
	// get "/foo" key's value
	log.Print("Getting '/foo' key value")
	resp, err = kapi.Get(context.Background(), "/foo", nil)
	if err != nil {
		log.Fatal(err)
	} else {
		// print common key info
		log.Printf("Get is done. Metadata is %q\n", resp)
		// print value
		log.Printf("%q key has %q value\n", resp.Node.Key, resp.Node.Value)
	}
}

Error Handling

etcd client might return three types of errors.

context error

Each API call has its first parameter as context. A context can be canceled or have an attached deadline. If the context is canceled or reaches its deadline, the responding context error will be returned no matter what internal errors the API call has already encountered.

cluster error

Each API call tries to send request to the cluster endpoints one by one until it successfully gets a response. If a requests to an endpoint fails, due to exceeding per request timeout or connection issues, the error will be added into a list of errors. If all possible endpoints fail, a cluster error that includes all encountered errors will be returned.

response error

If the response gets from the cluster is invalid, a plain string error will be returned. For example, it might be a invalid JSON error.

Here is the example code to handle client errors:

cfg := client.Config{Endpoints: []string{"http://etcd1:2379","http://etcd2:2379","http://etcd3:2379"}}
c, err := client.New(cfg)
if err != nil {
	log.Fatal(err)
}

kapi := client.NewKeysAPI(c)
resp, err := kapi.Set(ctx, "test", "bar", nil)
if err != nil {
	if err == context.Canceled {
		// ctx is canceled by another routine
	} else if err == context.DeadlineExceeded {
		// ctx is attached with a deadline and it exceeded
	} else if cerr, ok := err.(*client.ClusterError); ok {
		// process (cerr.Errors)
	} else {
		// bad cluster endpoints, which are not etcd servers
	}
}

Caveat

etcd/client prefers to use the same endpoint as long as the endpoint continues to work well. This saves socket resources, and improves efficiency for both client and server side. This preference doesn't remove consistency from the data consumed by the client because data replicated to each etcd member has already passed through the consensus process.
etcd/client does round-robin rotation on other available endpoints if the preferred endpoint isn't functioning properly. For example, if the member that etcd/client connects to is hard killed, etcd/client will fail on the first attempt with the killed member, and succeed on the second attempt with another member. If it fails to talk to all available endpoints, it will return all errors happened.
Default etcd/client cannot handle the case that the remote server is SIGSTOPed now. TCP keepalive mechanism doesn't help in this scenario because operating system may still send TCP keep-alive packets. Over time we'd like to improve this functionality, but solving this issue isn't high priority because a real-life case in which a server is stopped, but the connection is kept alive, hasn't been brought to our attention.
etcd/client cannot detect whether a member is healthy with watches and non-quorum read requests. If the member is isolated from the cluster, etcd/client may retrieve outdated data. Instead, users can either issue quorum read requests or monitor the /health endpoint for member health information.

Overview (`third_party/coreos-etcd/clientv3/README.md`)

etcd/clientv3

etcd/clientv3 is the official Go etcd client for v3.

Install

bash

go get github.com/coreos/etcd/clientv3

Get started

Create client using clientv3.New:

cli, err := clientv3.New(clientv3.Config{
	Endpoints:   []string{"localhost:2379", "localhost:22379", "localhost:32379"},
	DialTimeout: 5 * time.Second,
})
if err != nil {
	// handle error!
}
defer cli.Close()

etcd v3 uses gRPC for remote procedure calls. And clientv3 uses
grpc-go to connect to etcd. Make sure to close the client after using it.
If the client is not closed, the connection will have leaky goroutines. To specify client request timeout,
pass context.WithTimeout to APIs:

ctx, cancel := context.WithTimeout(context.Background(), timeout)
resp, err := cli.Put(ctx, "sample_key", "sample_value")
cancel()
if err != nil {
    // handle error!
}
// use the response

Error Handling

etcd client returns 2 types of errors:

context error: canceled or deadline exceeded.
gRPC error: see api/v3rpc/rpctypes.

Here is the example code to handle client errors:

resp, err := cli.Put(ctx, "", "")
if err != nil {
	switch err {
	case context.Canceled:
		log.Fatalf("ctx is canceled by another routine: %v", err)
	case context.DeadlineExceeded:
		log.Fatalf("ctx is attached with a deadline is exceeded: %v", err)
	case rpctypes.ErrEmptyKey:
		log.Fatalf("client-side error: %v", err)
	default:
		log.Fatalf("bad cluster endpoints, which are not etcd servers: %v", err)
	}
}

Metrics

The etcd client optionally exposes RPC metrics through go-grpc-prometheus. See the examples.

Namespacing

The namespace package provides clientv3 interface wrappers to transparently isolate client requests to a user-defined prefix.

Examples

More code examples can be found at GoDoc.

Overview (`third_party/coreos-etcd/cmd/README.md`)

cmd

This directory is meant to enforce vendoring for etcd binaries without polluting
the etcd client libraries with vendored dependencies.

Overview (`third_party/coreos-etcd/contrib/README.md`)

Contrib

Scripts and files which may be useful but aren't part of the core etcd project.

systemd - an example unit file for deploying etcd on systemd-based distributions
raftexample - an example distributed key-value store using raft
systemd/etcd2-backup-coreos - remote backup and restore procedures for etcd2 clusters on CoreOS Linux
systemd/etcd3-multinode - multi-node cluster setup with systemd

Overview (`third_party/coreos-etcd/etcdctl/README.md`)

etcdctl

etcdctl is a command line client for etcd.
Make sure to set environment variable ETCDCTL_API=3. For etcdctl v2, please check READMEv2.

Global flags (e.g., dial-timeout, --cacert, --cert, --key) can be set with environment variables:

text

ETCDCTL_DIAL_TIMEOUT=3s
ETCDCTL_CACERT=/tmp/ca.pem
ETCDCTL_CERT=/tmp/cert.pem
ETCDCTL_KEY=/tmp/key.pem

Prefix flag strings with ETCDCTL_, convert all letters to upper-case, and replace dash(-) with underscore(_).

Key-value commands

PUT [options] \ \

PUT assigns the specified value with the specified key. If key already holds a value, it is overwritten.

RPC: Put

Options

lease -- lease ID (in hexadecimal) to attach to the key.
prev-kv -- return the previous key-value pair before modification.
ignore-value -- updates the key using its current value.
ignore-lease -- updates the key using its current lease.

Output

OK

Examples

bash

./etcdctl put foo bar --lease=1234abcd
# OK
./etcdctl get foo
# foo
# bar
./etcdctl put foo --ignore-value # to detache lease
# OK

bash

./etcdctl put foo bar --lease=1234abcd
# OK
./etcdctl put foo bar1 --ignore-lease # to use existing lease 1234abcd
# OK
./etcdctl get foo
# foo
# bar1

bash

./etcdctl put foo bar1 --prev-kv
# OK
# foo
# bar
./etcdctl get foo
# foo
# bar1

Remarks

If <value> isn't given as command line argument, this command tries to read the value from standard input.

When <value> begins with '-', <value> is interpreted as a flag.
Insert '--' for workaround:

bash

./etcdctl put <key> -- <value>
./etcdctl put -- <key> <value>

GET [options] \ [range_end]

GET gets the key or a range of keys [key, range_end) if range-end is given.

RPC: Range

Options

hex -- print out key and value as hex encode string
limit -- maximum number of results
prefix -- get keys by matching prefix
order -- order of results; ASCEND or DESCEND
sort-by -- sort target; CREATE, KEY, MODIFY, VALUE, or VERSION
rev -- specify the kv revision
print-value-only -- print only value when used with write-out=simple
consistency -- Linearizable(l) or Serializable(s)
from-key -- Get keys that are greater than or equal to the given key using byte compare
keys-only -- Get only the keys

Output

<key>\n<value>\n<next_key>\n<next_value>...

Examples

First, populate etcd with some keys:

bash

./etcdctl put foo bar
# OK
./etcdctl put foo1 bar1
# OK
./etcdctl put foo2 bar2
# OK
./etcdctl put foo3 bar3
# OK

Get the key named foo:

bash

./etcdctl get foo
# foo
# bar

Get all keys:

bash

./etcdctl get --from-key ''
# foo
# bar
# foo1
# bar1
# foo2
# foo2
# foo3
# bar3

Get all keys with names greater than or equal to foo1:

bash

./etcdctl get --from-key foo1
# foo1
# bar1
# foo2
# bar2
# foo3
# bar3

Get keys with names greater than or equal to foo1 and less than foo3:

bash

./etcdctl get foo1 foo3
# foo1
# bar1
# foo2
# bar2

Remarks

If any key or value contains non-printable characters or control characters, simple formatted output can be ambiguous due to new lines. To resolve this issue, set --hex to hex encode all strings.

DEL [options] \ [range_end]

Removes the specified key or range of keys [key, range_end) if range-end is given.

RPC: DeleteRange

Options

prefix -- delete keys by matching prefix
prev-kv -- return deleted key-value pairs
from-key -- delete keys that are greater than or equal to the given key using byte compare

Output

Prints the number of keys that were removed in decimal if DEL succeeded.

Examples

bash

./etcdctl put foo bar
# OK
./etcdctl del foo
# 1
./etcdctl get foo

bash

./etcdctl put key val
# OK
./etcdctl del --prev-kv key
# 1
# key
# val
./etcdctl get key

bash

./etcdctl put a 123
# OK
./etcdctl put b 456
# OK
./etcdctl put z 789
# OK
./etcdctl del --from-key a
# 3
./etcdctl get --from-key a

bash

./etcdctl put zoo val
# OK
./etcdctl put zoo1 val1
# OK
./etcdctl put zoo2 val2
# OK
./etcdctl del --prefix zoo
# 3
./etcdctl get zoo2

TXN [options]

TXN reads multiple etcd requests from standard input and applies them as a single atomic transaction.
A transaction consists of list of conditions, a list of requests to apply if all the conditions are true, and a list of requests to apply if any condition is false.

RPC: Txn

Options

hex -- print out keys and values as hex encoded strings.
interactive -- input transaction with interactive prompting.

Input Format

ebnf

<Txn> ::= <CMP>* "\n" <THEN> "\n" <ELSE> "\n"
<CMP> ::= (<CMPCREATE>|<CMPMOD>|<CMPVAL>|<CMPVER>|<CMPLEASE>) "\n"
<CMPOP> ::= "<" | "=" | ">"
<CMPCREATE> := ("c"|"create")"("<KEY>")" <REVISION>
<CMPMOD> ::= ("m"|"mod")"("<KEY>")" <CMPOP> <REVISION>
<CMPVAL> ::= ("val"|"value")"("<KEY>")" <CMPOP> <VALUE>
<CMPVER> ::= ("ver"|"version")"("<KEY>")" <CMPOP> <VERSION>
<CMPLEASE> ::= "lease("<KEY>")" <CMPOP> <LEASE>
<THEN> ::= <OP>*
<ELSE> ::= <OP>*
<OP> ::= ((see put, get, del etcdctl command syntax)) "\n"
<KEY> ::= (%q formatted string)
<VALUE> ::= (%q formatted string)
<REVISION> ::= "\""[0-9]+"\""
<VERSION> ::= "\""[0-9]+"\""
<LEASE> ::= "\""[0-9]+\""

Output

SUCCESS if etcd processed the transaction success list, FAILURE if etcd processed the transaction failure list. Prints the output for each command in the executed request list, each separated by a blank line.

Examples

txn in interactive mode:

bash

./etcdctl txn -i
# compares:
mod("key1") > "0"

# success requests (get, put, delete):
put key1 "overwrote-key1"

# failure requests (get, put, delete):
put key1 "created-key1"
put key2 "some extra key"

# FAILURE

# OK

# OK

txn in non-interactive mode:

bash

./etcdctl txn <<<'mod("key1") > "0"

put key1 "overwrote-key1"

put key1 "created-key1"
put key2 "some extra key"

'

# FAILURE

# OK

# OK

COMPACTION [options] \

COMPACTION discards all etcd event history prior to a given revision. Since etcd uses a multiversion concurrency control
model, it preserves all key updates as event history. When the event history up to some revision is no longer needed,
all superseded keys may be compacted away to reclaim storage space in the etcd backend database.

RPC: Compact

Options

physical -- 'true' to wait for compaction to physically remove all old revisions

Output

Prints the compacted revision.

Example

bash

./etcdctl compaction 1234
# compacted revision 1234

WATCH [options] [key or prefix] [range_end] [--] [exec-command arg1 arg2 ...]

Watch watches events stream on keys or prefixes, [key or prefix, range_end) if range-end is given. The watch command runs until it encounters an error or is terminated by the user. If range_end is given, it must be lexicographically greater than key or "\x00".

RPC: Watch

Options

hex -- print out key and value as hex encode string
interactive -- begins an interactive watch session
prefix -- watch on a prefix if prefix is set.
prev-kv -- get the previous key-value pair before the event happens.
rev -- the revision to start watching. Specifying a revision is useful for observing past events.

Input format

Input is only accepted for interactive mode.

text

watch [options] <key or prefix>\n

Output

<event>[\n<old_key>\n<old_value>]\n<key>\n<value>\n<event>\n<next_key>\n<next_value>\n...

Examples

Non-interactive

bash

./etcdctl watch foo
# PUT
# foo
# bar

bash

ETCDCTL_WATCH_KEY=foo ./etcdctl watch
# PUT
# foo
# bar

Receive events and execute echo watch event received:

bash

./etcdctl watch foo -- echo watch event received
# PUT
# foo
# bar
# watch event received

Watch response is set via ETCD_WATCH_* environmental variables:

bash

./etcdctl watch foo -- sh -c "env | grep ETCD_WATCH_"

# PUT
# foo
# bar
# ETCD_WATCH_REVISION=11
# ETCD_WATCH_KEY="foo"
# ETCD_WATCH_EVENT_TYPE="PUT"
# ETCD_WATCH_VALUE="bar"

Watch with environmental variables and execute echo watch event received:

bash

export ETCDCTL_WATCH_KEY=foo
./etcdctl watch -- echo watch event received
# PUT
# foo
# bar
# watch event received

bash

export ETCDCTL_WATCH_KEY=foo
export ETCDCTL_WATCH_RANGE_END=foox
./etcdctl watch -- echo watch event received
# PUT
# fob
# bar
# watch event received

Interactive

bash

./etcdctl watch -i
watch foo
watch foo
# PUT
# foo
# bar
# PUT
# foo
# bar

Receive events and execute echo watch event received:

bash

./etcdctl watch -i
watch foo -- echo watch event received
# PUT
# foo
# bar
# watch event received

Watch with environmental variables and execute echo watch event received:

bash

export ETCDCTL_WATCH_KEY=foo
./etcdctl watch -i
watch -- echo watch event received
# PUT
# foo
# bar
# watch event received

bash

export ETCDCTL_WATCH_KEY=foo
export ETCDCTL_WATCH_RANGE_END=foox
./etcdctl watch -i
watch -- echo watch event received
# PUT
# fob
# bar
# watch event received

LEASE \

LEASE provides commands for key lease management.

LEASE GRANT \

LEASE GRANT creates a fresh lease with a server-selected time-to-live in seconds
greater than or equal to the requested TTL value.

RPC: LeaseGrant

Output

Prints a message with the granted lease ID.

Example

bash

./etcdctl lease grant 10
# lease 32695410dcc0ca06 granted with TTL(10s)

LEASE REVOKE \

LEASE REVOKE destroys a given lease, deleting all attached keys.

RPC: LeaseRevoke

Output

Prints a message indicating the lease is revoked.

Example

bash

./etcdctl lease revoke 32695410dcc0ca06
# lease 32695410dcc0ca06 revoked

LEASE TIMETOLIVE \ [options]

LEASE TIMETOLIVE retrieves the lease information with the given lease ID.

RPC: LeaseTimeToLive

Options

keys -- Get keys attached to this lease

Output

Prints lease information.

Example

bash

./etcdctl lease grant 500
# lease 2d8257079fa1bc0c granted with TTL(500s)

./etcdctl put foo1 bar --lease=2d8257079fa1bc0c
# OK

./etcdctl put foo2 bar --lease=2d8257079fa1bc0c
# OK

./etcdctl lease timetolive 2d8257079fa1bc0c
# lease 2d8257079fa1bc0c granted with TTL(500s), remaining(481s)

./etcdctl lease timetolive 2d8257079fa1bc0c --keys
# lease 2d8257079fa1bc0c granted with TTL(500s), remaining(472s), attached keys([foo2 foo1])

./etcdctl lease timetolive 2d8257079fa1bc0c --write-out=json
# {"cluster_id":17186838941855831277,"member_id":4845372305070271874,"revision":3,"raft_term":2,"id":3279279168933706764,"ttl":465,"granted-ttl":500,"keys":null}

./etcdctl lease timetolive 2d8257079fa1bc0c --write-out=json --keys
# {"cluster_id":17186838941855831277,"member_id":4845372305070271874,"revision":3,"raft_term":2,"id":3279279168933706764,"ttl":459,"granted-ttl":500,"keys":["Zm9vMQ==","Zm9vMg=="]}

./etcdctl lease timetolive 2d8257079fa1bc0c
# lease 2d8257079fa1bc0c already expired

LEASE LIST

LEASE LIST lists all active leases.

RPC: LeaseLeases

Output

Prints a message with a list of active leases.

Example

bash

./etcdctl lease grant 10
# lease 32695410dcc0ca06 granted with TTL(10s)

./etcdctl lease list
32695410dcc0ca06

LEASE KEEP-ALIVE \

LEASE KEEP-ALIVE periodically refreshes a lease so it does not expire.

RPC: LeaseKeepAlive

Output

Prints a message for every keep alive sent or prints a message indicating the lease is gone.

Example

bash

./etcdctl lease keep-alive 32695410dcc0ca0
# lease 32695410dcc0ca0 keepalived with TTL(100)
# lease 32695410dcc0ca0 keepalived with TTL(100)
# lease 32695410dcc0ca0 keepalived with TTL(100)
...

Cluster maintenance commands

MEMBER \

MEMBER provides commands for managing etcd cluster membership.

MEMBER ADD \ [options]

MEMBER ADD introduces a new member into the etcd cluster as a new peer.

RPC: MemberAdd

Options

peer-urls -- comma separated list of URLs to associate with the new member.

Output

Prints the member ID of the new member and the cluster ID.

Example

bash

./etcdctl member add newMember --peer-urls=https://127.0.0.1:12345

Member ced000fda4d05edf added to cluster 8c4281cc65c7b112

ETCD_NAME="newMember"
ETCD_INITIAL_CLUSTER="newMember=https://127.0.0.1:12345,default=http://10.0.0.30:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

MEMBER UPDATE \ [options]

MEMBER UPDATE sets the peer URLs for an existing member in the etcd cluster.

RPC: MemberUpdate

Options

peer-urls -- comma separated list of URLs to associate with the updated member.

Output

Prints the member ID of the updated member and the cluster ID.

Example

bash

./etcdctl member update 2be1eb8f84b7f63e --peer-urls=https://127.0.0.1:11112
# Member 2be1eb8f84b7f63e updated in cluster ef37ad9dc622a7c4

MEMBER REMOVE \

MEMBER REMOVE removes a member of an etcd cluster from participating in cluster consensus.

RPC: MemberRemove

Output

Prints the member ID of the removed member and the cluster ID.

Example

bash

./etcdctl member remove 2be1eb8f84b7f63e
# Member 2be1eb8f84b7f63e removed from cluster ef37ad9dc622a7c4

MEMBER LIST

MEMBER LIST prints the member details for all members associated with an etcd cluster.

RPC: MemberList.

Output

Prints a humanized table of the member IDs, statuses, names, peer addresses, and client addresses.

Examples

bash

./etcdctl member list
# 8211f1d0f64f3269, started, infra1, http://127.0.0.1:12380, http://127.0.0.1:2379
# 91bc3c398fb3c146, started, infra2, http://127.0.0.1:22380, http://127.0.0.1:22379
# fd422379fda50e48, started, infra3, http://127.0.0.1:32380, http://127.0.0.1:32379

bash

./etcdctl -w json member list
# {"header":{"cluster_id":17237436991929493444,"member_id":9372538179322589801,"raft_term":2},"members":[{"ID":9372538179322589801,"name":"infra1","peerURLs":["http://127.0.0.1:12380"],"clientURLs":["http://127.0.0.1:2379"]},{"ID":10501334649042878790,"name":"infra2","peerURLs":["http://127.0.0.1:22380"],"clientURLs":["http://127.0.0.1:22379"]},{"ID":18249187646912138824,"name":"infra3","peerURLs":["http://127.0.0.1:32380"],"clientURLs":["http://127.0.0.1:32379"]}]}

bash

./etcdctl -w table member list
+------------------+---------+--------+------------------------+------------------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+------------------+---------+--------+------------------------+------------------------+
| 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:12380 | http://127.0.0.1:2379  |
| 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
| fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
+------------------+---------+--------+------------------------+------------------------+

ENDPOINT \

ENDPOINT provides commands for querying individual endpoints.

Options

cluster -- fetch and use all endpoints from the etcd cluster member list

ENDPOINT HEALTH

ENDPOINT HEALTH checks the health of the list of endpoints with respect to cluster. An endpoint is unhealthy
when it cannot participate in consensus with the rest of the cluster.

Output

If an endpoint can participate in consensus, prints a message indicating the endpoint is healthy. If an endpoint fails to participate in consensus, prints a message indicating the endpoint is unhealthy.

Example

Check the default endpoint's health:

bash

./etcdctl endpoint health
# 127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.095242ms

Check all endpoints for the cluster associated with the default endpoint:

bash

./etcdctl endpoint --cluster health
# http://127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.060091ms
# http://127.0.0.1:22379 is healthy: successfully committed proposal: took = 903.138µs
# http://127.0.0.1:32379 is healthy: successfully committed proposal: took = 1.113848ms

ENDPOINT STATUS

ENDPOINT STATUS queries the status of each endpoint in the given endpoint list.

Output

Simple format

Prints a humanized table of each endpoint URL, ID, version, database size, leadership status, raft term, and raft status.

JSON format

Prints a line of JSON encoding each endpoint URL, ID, version, database size, leadership status, raft term, and raft status.

Examples

Get the status for the default endpoint:

bash

./etcdctl endpoint status
# 127.0.0.1:2379, 8211f1d0f64f3269, 3.0.0, 25 kB, false, 2, 63

Get the status for the default endpoint as JSON:

bash

./etcdctl -w json endpoint status
# [{"Endpoint":"127.0.0.1:2379","Status":{"header":{"cluster_id":17237436991929493444,"member_id":9372538179322589801,"revision":2,"raft_term":2},"version":"3.0.0","dbSize":24576,"leader":18249187646912138824,"raftIndex":32623,"raftTerm":2}}]

Get the status for all endpoints in the cluster associated with the default endpoint:

bash

./etcdctl -w table endpoint --cluster status
+------------------------+------------------+----------------+---------+-----------+-----------+------------+
|        ENDPOINT        |        ID        |    VERSION     | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------------+------------------+----------------+---------+-----------+-----------+------------+
| http://127.0.0.1:2379  | 8211f1d0f64f3269 | 3.2.0-rc.1+git |   25 kB |     false |         2 |          8 |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 | 3.2.0-rc.1+git |   25 kB |     false |         2 |          8 |
| http://127.0.0.1:32379 | fd422379fda50e48 | 3.2.0-rc.1+git |   25 kB |      true |         2 |          8 |
+------------------------+------------------+----------------+---------+-----------+-----------+------------+

ENDPOINT HASHKV

ENDPOINT HASHKV fetches the hash of the key-value store of an endpoint.

Output

Simple format

Prints a humanized table of each endpoint URL and KV history hash.

JSON format

Prints a line of JSON encoding each endpoint URL and KV history hash.

Examples

Get the hash for the default endpoint:

bash

./etcdctl endpoint hashkv
# 127.0.0.1:2379, 1084519789

Get the status for the default endpoint as JSON:

bash

./etcdctl -w json endpoint hashkv
# [{"Endpoint":"127.0.0.1:2379","Hash":{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":1,"raft_term":3},"hash":1084519789,"compact_revision":-1}}]

Get the status for all endpoints in the cluster associated with the default endpoint:

bash

./etcdctl -w table endpoint --cluster hashkv
+------------------------+------------+
|        ENDPOINT        |    HASH    |
+------------------------+------------+
| http://127.0.0.1:12379 | 1084519789 |
| http://127.0.0.1:22379 | 1084519789 |
| http://127.0.0.1:32379 | 1084519789 |
+------------------------+------------+

ALARM \

Provides alarm related commands

ALARM DISARM

alarm disarm Disarms all alarms

RPC: Alarm

Output

alarm:<alarm type> if alarm is present and disarmed.

Examples

bash

./etcdctl alarm disarm

If NOSPACE alarm is present:

bash

./etcdctl alarm disarm
# alarm:NOSPACE

ALARM LIST

alarm list lists all alarms.

RPC: Alarm

Output

alarm:<alarm type> if alarm is present, empty string if no alarms present.

Examples

bash

./etcdctl alarm list

If NOSPACE alarm is present:

bash

./etcdctl alarm list
# alarm:NOSPACE

DEFRAG [options]

DEFRAG defragments the backend database file for a set of given endpoints while etcd is running, or directly defragments an etcd data directory while etcd is not running. When an etcd member reclaims storage space from deleted and compacted keys, the space is kept in a free list and the database file remains the same size. By defragmenting the database, the etcd member releases this free space back to the file system.

Note that defragmentation to a live member blocks the system from reading and writing data while rebuilding its states.

Note that defragmentation request does not get replicated over cluster. That is, the request is only applied to the local node. Specify all members in --endpoints flag.

Options

data-dir -- Optional. If present, defragments a data directory not in use by etcd.

Output

For each endpoints, prints a message indicating whether the endpoint was successfully defragmented.

Example

bash

./etcdctl --endpoints=localhost:2379,badendpoint:2379 defrag
# Finished defragmenting etcd member[localhost:2379]
# Failed to defragment etcd member[badendpoint:2379] (grpc: timed out trying to connect)

To defragment a data directory directly, use the --data-dir flag:

bash

# Defragment while etcd is not running
./etcdctl defrag --data-dir default.etcd
# success (exit status 0)
# Error: cannot open database at default.etcd/member/snap/db

Remarks

DEFRAG returns a zero exit code only if it succeeded defragmenting all given endpoints.

SNAPSHOT \

SNAPSHOT provides commands to restore a snapshot of a running etcd server into a fresh cluster.

SNAPSHOT SAVE \

SNAPSHOT SAVE writes a point-in-time snapshot of the etcd backend database to a file.

Output

The backend snapshot is written to the given file path.

Example

Save a snapshot to "snapshot.db":

text

./etcdctl snapshot save snapshot.db

SNAPSHOT RESTORE [options] \

SNAPSHOT RESTORE creates an etcd data directory for an etcd cluster member from a backend database snapshot and a new cluster configuration. Restoring the snapshot into each member for a new cluster configuration will initialize a new etcd cluster preloaded by the snapshot data.

Options

The snapshot restore options closely resemble to those used in the etcd command for defining a cluster.

data-dir -- Path to the data directory. Uses <name>.etcd if none given.
wal-dir -- Path to the WAL directory. Uses data directory if none given.
initial-cluster -- The initial cluster configuration for the restored etcd cluster.
initial-cluster-token -- Initial cluster token for the restored etcd cluster.
initial-advertise-peer-urls -- List of peer URLs for the member being restored.
name -- Human-readable name for the etcd cluster member being restored.
skip-hash-check -- Ignore snapshot integrity hash value (required if copied from data directory)

Output

A new etcd data directory initialized with the snapshot.

Example

Save a snapshot, restore into a new 3 node cluster, and start the cluster:

text

./etcdctl snapshot save snapshot.db

# restore members
bin/etcdctl snapshot restore snapshot.db --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://127.0.0.1:12380  --name sshot1 --initial-cluster 'sshot1=http://127.0.0.1:12380,sshot2=http://127.0.0.1:22380,sshot3=http://127.0.0.1:32380'
bin/etcdctl snapshot restore snapshot.db --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://127.0.0.1:22380  --name sshot2 --initial-cluster 'sshot1=http://127.0.0.1:12380,sshot2=http://127.0.0.1:22380,sshot3=http://127.0.0.1:32380'
bin/etcdctl snapshot restore snapshot.db --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://127.0.0.1:32380  --name sshot3 --initial-cluster 'sshot1=http://127.0.0.1:12380,sshot2=http://127.0.0.1:22380,sshot3=http://127.0.0.1:32380'

# launch members
bin/etcd --name sshot1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 &
bin/etcd --name sshot2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 &
bin/etcd --name sshot3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 &

SNAPSHOT STATUS \

SNAPSHOT STATUS lists information about a given backend database snapshot file.

Output

Simple format

Prints a humanized table of the database hash, revision, total keys, and size.

JSON format

Prints a line of JSON encoding the database hash, revision, total keys, and size.

Examples

bash

./etcdctl snapshot status file.db
# cf1550fb, 3, 3, 25 kB

bash

./etcdctl -write-out=json snapshot status file.db
# {"hash":3474280699,"revision":3,"totalKey":3,"totalSize":24576}

bash

./etcdctl -write-out=table snapshot status file.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| cf1550fb |        3 |          3 | 25 kB      |
+----------+----------+------------+------------+

MOVE-LEADER \

MOVE-LEADER transfers leadership from the leader to another member in the cluster.

Example

bash

# to choose transferee
transferee_id=$(./etcdctl \
  --endpoints localhost:12379,localhost:22379,localhost:32379 \
  endpoint status | grep -m 1 "false" | awk -F', ' '{print $2}')
echo ${transferee_id}
# c89feb932daef420

# endpoints should include leader node
./etcdctl --endpoints ${transferee_ep} move-leader ${transferee_id}
# Error:  no leader endpoint given at [localhost:22379 localhost:32379]

# request to leader with target node ID
./etcdctl --endpoints ${leader_ep} move-leader ${transferee_id}
# Leadership transferred from 45ddc0e800e20b93 to c89feb932daef420

Concurrency commands

LOCK [options] \ [command arg1 arg2 ...]

LOCK acquires a distributed named mutex with a given name. Once the lock is acquired, it will be held until etcdctl is terminated.

Options

ttl - time out in seconds of lock session.

Output

Once the lock is acquired, the result for the GET on the unique lock holder key is displayed.

If a command is given, it will be launched with environment variables ETCD_LOCK_KEY and ETCD_LOCK_REV set to the lock's holder key and revision.

Example

Acquire lock with standard output display:

bash

./etcdctl lock mylock
# mylock/1234534535445

Acquire lock and execute echo lock acquired:

bash

./etcdctl lock mylock echo lock acquired
# lock acquired

Remarks

LOCK returns a zero exit code only if it is terminated by a signal and releases the lock.

If LOCK is abnormally terminated or fails to contact the cluster to release the lock, the lock will remain held until the lease expires. Progress may be delayed by up to the default lease length of 60 seconds.

ELECT [options] \ [proposal]

ELECT participates on a named election. A node announces its candidacy in the election by providing
a proposal value. If a node wishes to observe the election, ELECT listens for new leaders values.
Whenever a leader is elected, its proposal is given as output.

Options

listen -- observe the election.

Output

If a candidate, ELECT displays the GET on the leader key once the node is elected election.
If observing, ELECT streams the result for a GET on the leader key for the current election and all future elections.

Example

bash

./etcdctl elect myelection foo
# myelection/1456952310051373265
# foo

Remarks

ELECT returns a zero exit code only if it is terminated by a signal and can revoke its candidacy or leadership, if any.

If a candidate is abnormally terminated, election rogress may be delayed by up to the default lease length of 60 seconds.

Authentication commands

AUTH \

auth enable activates authentication on an etcd cluster and auth disable deactivates. When authentication is enabled, etcd checks all requests for appropriate authorization.

RPC: AuthEnable/AuthDisable

Output

Authentication Enabled.

Examples

bash

./etcdctl user add root
# Password of root:#type password for root
# Type password of root again for confirmation:#re-type password for root
# User root created
./etcdctl user grant-role root root
# Role root is granted to user root
./etcdctl user get root
# User: root
# Roles: root
./etcdctl role add root
# Role root created
./etcdctl role get root
# Role root
# KV Read:
# KV Write:
./etcdctl auth enable
# Authentication Enabled

ROLE \

ROLE is used to specify differnt roles which can be assigned to etcd user(s).

ROLE ADD \

role add creates a role.

RPC: RoleAdd

Output

Role <role name> created.

Examples

bash

./etcdctl --user=root:123 role add myrole
# Role myrole created

ROLE GET \

role get lists detailed role information.

RPC: RoleGet

Output

Detailed role information.

Examples

bash

./etcdctl --user=root:123 role get myrole
# Role myrole
# KV Read:
# foo
# KV Write:
# foo

ROLE DELETE \

role delete deletes a role.

RPC: RoleDelete

Output

Role <role name> deleted.

Examples

bash

./etcdctl --user=root:123 role delete myrole
# Role myrole deleted

ROLE LIST \

role list lists all roles in etcd.

RPC: RoleList

Output

A role per line.

Examples

bash

./etcdctl --user=root:123 role list
# roleA
# roleB
# myrole

ROLE GRANT-PERMISSION [options] \ \ \ [endkey]

role grant-permission grants a key to a role.

RPC: RoleGrantPermission

Options

from-key -- grant a permission of keys that are greater than or equal to the given key using byte compare
prefix -- grant a prefix permission

Output

Role <role name> updated.

Examples

Grant read and write permission on the key foo to role myrole:

bash

./etcdctl --user=root:123 role grant-permission myrole readwrite foo
# Role myrole updated

Grant read permission on the wildcard key pattern foo/* to role myrole:

bash

./etcdctl --user=root:123 role grant-permission --prefix myrole readwrite foo/
# Role myrole updated

ROLE REVOKE-PERMISSION \ \ \ [endkey]

role revoke-permission revokes a key from a role.

RPC: RoleRevokePermission

Options

from-key -- revoke a permission of keys that are greater than or equal to the given key using byte compare
prefix -- revoke a prefix permission

Output

Permission of key <key> is revoked from role <role name> for single key. Permission of range [<key>, <endkey>) is revoked from role <role name> for a key range. Exit code is zero.

Examples

bash

./etcdctl --user=root:123 role revoke-permission myrole foo
# Permission of key foo is revoked from role myrole

USER \

USER provides commands for managing users of etcd.

USER ADD \ [options]

user add creates a user.

RPC: UserAdd

Options

interactive -- Read password from stdin instead of interactive terminal

Output

User created.

Examples

bash

./etcdctl --user=root:123 user add myuser
# Password of myuser: #type password for my user
# Type password of myuser again for confirmation:#re-type password for my user
# User myuser created

USER GET \ [options]

user get lists detailed user information.

RPC: UserGet

Options

detail -- Show permissions of roles granted to the user

Output

Detailed user information.

Examples

bash

./etcdctl --user=root:123 user get myuser
# User: myuser
# Roles:

USER DELETE \

user delete deletes a user.

RPC: UserDelete

Output

User deleted.

Examples

bash

./etcdctl --user=root:123 user delete myuser
# User myuser deleted

USER LIST

user list lists detailed user information.

RPC: UserList

Output

List of users, one per line.

Examples

bash

./etcdctl --user=root:123 user list
# user1
# user2
# myuser

USER PASSWD \ [options]

user passwd changes a user's password.

RPC: UserChangePassword

Options

interactive -- if true, read password in interactive terminal

Output

Password updated.

Examples

bash

./etcdctl --user=root:123 user passwd myuser
# Password of myuser: #type new password for my user
# Type password of myuser again for confirmation: #re-type the new password for my user
# Password updated

USER GRANT-ROLE \ \

user grant-role grants a role to a user

RPC: UserGrantRole

Output

Role <role name> is granted to user .

Examples

bash

./etcdctl --user=root:123 user grant-role userA roleA
# Role roleA is granted to user userA

USER REVOKE-ROLE \ \

user revoke-role revokes a role from a user

RPC: UserRevokeRole

Output

Role <role name> is revoked from user .

Examples

bash

./etcdctl --user=root:123 user revoke-role userA roleA
# Role roleA is revoked from user userA

Utility commands

MAKE-MIRROR [options] \

make-mirror mirrors a key prefix in an etcd cluster to a destination etcd cluster.

Options

dest-cacert -- TLS certificate authority file for destination cluster
dest-cert -- TLS certificate file for destination cluster
dest-key -- TLS key file for destination cluster
prefix -- The key-value prefix to mirror
dest-prefix -- The destination prefix to mirror a prefix to a different prefix in the destination cluster
no-dest-prefix -- Mirror key-values to the root of the destination cluster
dest-insecure-transport -- Disable transport security for client connections

Output

The approximate total number of keys transferred to the destination cluster, updated every 30 seconds.

Examples

text

./etcdctl make-mirror mirror.example.com:2379
# 10
# 18

MIGRATE [options]

Migrates keys in a v2 store to a v3 mvcc store. Users should run migration command for all members in the cluster.

Options

data-dir -- Path to the data directory
wal-dir -- Path to the WAL directory
transformer -- Path to the user-provided transformer program (default if not provided)

Output

No output on success.

Default transformer

If user does not provide a transformer program, migrate command will use the default transformer. The default transformer transforms storev2 formatted keys into mvcc formatted keys according to the following Go program:

func transform(n *storev2.Node) *mvccpb.KeyValue {
	if n.Dir {
		return nil
	}
	kv := &mvccpb.KeyValue{
		Key:            []byte(n.Key),
		Value:          []byte(n.Value),
		CreateRevision: int64(n.CreatedIndex),
		ModRevision:    int64(n.ModifiedIndex),
		Version:        1,
	}
	return kv
}

User-provided transformer

Users can provide a customized 1:n transformer function that transforms a key from the v2 store to any number of keys in the mvcc store. The migration program writes JSON formatted v2 store keys to the transformer program's stdin, reads protobuf formatted mvcc keys back from the transformer program's stdout, and finishes migration by saving the transformed keys into the mvcc store.

The provided transformer should read until EOF and flush the stdout before exiting to ensure data integrity.

Example

text

./etcdctl migrate --data-dir=/var/etcd --transformer=k8s-transformer
# finished transforming keys

VERSION

Prints the version of etcdctl.

Output

Prints etcd version and API version.

Examples

bash

./etcdctl version
# etcdctl version: 3.1.0-alpha.0+git
# API version: 3.1

Exit codes

For all commands, a successful execution return a zero exit code. All failures will return non-zero exit codes.

Output formats

All commands accept an output format by setting -w or --write-out. All commands default to the "simple" output format, which is meant to be human-readable. The simple format is listed in each command's Output description since it is customized for each command. If a command has a corresponding RPC, it will respect all output formats.

If a command fails, returning a non-zero exit code, an error string will be written to standard error regardless of output format.

Simple

A format meant to be easy to parse and human-readable. Specific to each command.

JSON

The JSON encoding of the command's RPC response. Since etcd's RPCs use byte strings, the JSON output will encode keys and values in base64.

Some commands without an RPC also support JSON; see the command's Output description.

Protobuf

The protobuf encoding of the command's RPC response. If an RPC is streaming, the stream messages will be concetenated. If an RPC is not given for a command, the protobuf output is not defined.

Fields

An output format similar to JSON but meant to parse with coreutils. For an integer field named Field, it writes a line in the format "Field" : %d where %d is go's integer formatting. For byte array fields, it writes "Field" : %q where %q is go's quoted string formatting (e.g., []byte{'a', '\n'} is written as "a\n").

Compatibility Support

etcdctl is still in its early stage. We try out best to ensure fully compatible releases, however we might break compatibility to fix bugs or improve commands. If we intend to release a version of etcdctl with backward incompatibilities, we will provide notice prior to release and have instructions on how to upgrade.

Input Compatibility

Input includes the command name, its flags, and its arguments. We ensure backward compatibility of the input of normal commands in non-interactive mode.

Output Compatibility

Output includes output from etcdctl and its exit code. etcdctl provides simple output format by default.
We ensure compatibility for the simple output format of normal commands in non-interactive mode. Currently, we do not ensure
backward compatibility for JSON format and the format in non-interactive mode. Currently, we do not ensure backward compatibility of utility commands.

TODO: compatibility with etcd server

Overview (`third_party/coreos-etcd/functional/README.md`)

etcd Functional Testing

functional verifies the correct behavior of etcd under various system and network malfunctions. It sets up an etcd cluster under high pressure loads and continuously injects failures into the cluster. Then it expects the etcd cluster to recover within a few seconds. This has been extremely helpful to find critical bugs.

See rpcpb.Case for all failure cases.

See functional.yaml for an example configuration.

Run locally

bash

PASSES=functional ./test

Run with Docker

bash

pushd ..
make build-docker-functional
popd

And run example scripts.

bash

# run 3 agents for 3-node local etcd cluster
./scripts/docker-local-agent.sh 1
./scripts/docker-local-agent.sh 2
./scripts/docker-local-agent.sh 3

# to run only 1 tester round
./scripts/docker-local-tester.sh

etcd Proxy

Proxy layer that simulates various network conditions.

Test locally

bash

$ ./build
$ ./bin/etcd

$ make build-functional

$ ./bin/etcd-proxy --help
$ ./bin/etcd-proxy --from localhost:23790 --to localhost:2379 --http-port 2378 --verbose

$ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:2379 put foo bar
$ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:23790 put foo bar

Proxy overhead per request is under 500μs

text

/* Detailed source-code truncated for AI context efficiency. */

Delay client transmit

bash

$ curl -L http://localhost:2378/delay-tx -X PUT \
  -d "latency=5s&random-variable=100ms"
# added send latency 5s±100ms (current latency 4.92143955s)

$ curl -L http://localhost:2378/delay-tx
# current send latency 4.92143955s

$ ETCDCTL_API=3 ./bin/etcdctl \
  --endpoints localhost:23790 \
  --command-timeout=3s \
  put foo bar
# Error: context deadline exceeded

$ curl -L http://localhost:2378/delay-tx -X DELETE
# removed latency 4.92143955s

$ curl -L http://localhost:2378/delay-tx
# current send latency 0s

$ ETCDCTL_API=3 ./bin/etcdctl \
  --endpoints localhost:23790 \
  --command-timeout=3s \
  put foo bar
# OK

Pause client transmit

bash

$ curl -L http://localhost:2378/pause-tx -X PUT
# paused forwarding [tcp://localhost:23790 -> tcp://localhost:2379]

$ ETCDCTL_API=3 ./bin/etcdctl \
  --endpoints localhost:23790 \
  put foo bar
# Error: context deadline exceeded

$ curl -L http://localhost:2378/pause-tx -X DELETE
# unpaused forwarding [tcp://localhost:23790 -> tcp://localhost:2379]

Drop client packets

bash

$ curl -L http://localhost:2378/blackhole-tx -X PUT
# blackholed; dropping packets [tcp://localhost:23790 -> tcp://localhost:2379]

$ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:23790 put foo bar
# Error: context deadline exceeded

$ curl -L http://localhost:2378/blackhole-tx -X DELETE
# unblackholed; restart forwarding [tcp://localhost:23790 -> tcp://localhost:2379]

Trigger leader election

bash

$ ./build
$ make build-functional

$ rm -rf /tmp/etcd-proxy-data.s*
$ goreman -f ./functional/Procfile-proxy start

$ ETCDCTL_API=3 ./bin/etcdctl \
  --endpoints localhost:13790,localhost:23790,localhost:33790 \
  member list

# isolate s1 when s1 is the current leader
$ curl -L http://localhost:1381/blackhole-tx -X PUT
$ curl -L http://localhost:1381/blackhole-rx -X PUT
# s1 becomes follower after election timeout

Overview (`third_party/coreos-etcd/pkg/README.md`)

pkg/ is a collection of utility packages used by etcd without being specific to etcd itself. A package belongs here
only if it could possibly be moved out into its own repository in the future.

Overview (`third_party/coreos-etcd/raft/README.md`)

Raft library

Raft is a protocol with which a cluster of nodes can maintain a replicated state machine.
The state machine is kept in sync through the use of a replicated log.
For more details on Raft, see "In Search of an Understandable Consensus Algorithm"
(https://ramcloud.stanford.edu/raft.pdf) by Diego Ongaro and John Ousterhout.

This Raft library is stable and feature complete. As of 2016, it is the most widely used Raft library in production, serving tens of thousands clusters each day. It powers distributed systems such as etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, and more.

Most Raft implementations have a monolithic design, including storage handling, messaging serialization, and network transport. This library instead follows a minimalistic design philosophy by only implementing the core raft algorithm. This minimalism buys flexibility, determinism, and performance.

To keep the codebase small as well as provide flexibility, the library only implements the Raft algorithm; both network and disk IO are left to the user. Library users must implement their own transportation layer for message passing between Raft peers over the wire. Similarly, users must implement their own storage layer to persist the Raft log and state.

In order to easily test the Raft library, its behavior should be deterministic. To achieve this determinism, the library models Raft as a state machine. The state machine takes a Message as input. A message can either be a local timer update or a network message sent from a remote peer. The state machine's output is a 3-tuple {[]Messages, []LogEntries, NextState} consisting of an array of Messages, log entries, and Raft state changes. For state machines with the same state, the same state machine input should always generate the same state machine output.

A simple example application, raftexample, is also available to help illustrate how to use this package in practice: https://github.com/coreos/etcd/tree/master/contrib/raftexample

Features

This raft implementation is a full feature implementation of Raft protocol. Features includes:

Leader election
Log replication
Log compaction
Membership changes
Leadership transfer extension
Efficient linearizable read-only queries served by both the leader and followers
- leader checks with quorum and bypasses Raft log before processing read-only queries
- followers asks leader to get a safe read index before processing read-only queries
More efficient lease-based linearizable read-only queries served by both the leader and followers
- leader bypasses Raft log and processing read-only queries locally
- followers asks leader to get a safe read index before processing read-only queries
- this approach relies on the clock of the all the machines in raft group

This raft implementation also includes a few optional enhancements:

Optimistic pipelining to reduce log replication latency
Flow control for log replication
Batching Raft messages to reduce synchronized network I/O calls
Batching log entries to reduce disk synchronized I/O
Writing to leader's disk in parallel
Internal proposal redirection from followers to leader
Automatic stepping down when the leader loses quorum

Notable Users

cockroachdb A Scalable, Survivable, Strongly-Consistent SQL Database
dgraph A Scalable, Distributed, Low Latency, High Throughput Graph Database
etcd A distributed reliable key-value store
tikv A Distributed transactional key value database powered by Rust and Raft
swarmkit A toolkit for orchestrating distributed systems at any scale.
chain core Software for operating permissioned, multi-asset blockchain networks

Usage

The primary object in raft is a Node. Either start a Node from scratch using raft.StartNode or start a Node from some initial state using raft.RestartNode.

To start a three-node cluster

storage := raft.NewMemoryStorage()
  c := &Config{
    ID:              0x01,
    ElectionTick:    10,
    HeartbeatTick:   1,
    Storage:         storage,
    MaxSizePerMsg:   4096,
    MaxInflightMsgs: 256,
  }
  // Set peer list to the other nodes in the cluster.
  // Note that they need to be started separately as well.
  n := raft.StartNode(c, []raft.Peer{{ID: 0x02}, {ID: 0x03}})

Start a single node cluster, like so:

// Create storage and config as shown above.
  // Set peer list to itself, so this node can become the leader of this single-node cluster.
  peers := []raft.Peer{{ID: 0x01}}
  n := raft.StartNode(c, peers)

To allow a new node to join this cluster, do not pass in any peers. First, add the node to the existing cluster by calling ProposeConfChange on any existing node inside the cluster. Then, start the node with an empty peer list, like so:

// Create storage and config as shown above.
  n := raft.StartNode(c, nil)

To restart a node from previous state:

storage := raft.NewMemoryStorage()

  // Recover the in-memory storage from persistent snapshot, state and entries.
  storage.ApplySnapshot(snapshot)
  storage.SetHardState(state)
  storage.Append(entries)

  c := &Config{
    ID:              0x01,
    ElectionTick:    10,
    HeartbeatTick:   1,
    Storage:         storage,
    MaxSizePerMsg:   4096,
    MaxInflightMsgs: 256,
  }

  // Restart raft without peer information.
  // Peer information is already included in the storage.
  n := raft.RestartNode(c)

After creating a Node, the user has a few responsibilities:

First, read from the Node.Ready() channel and process the updates it contains. These steps may be performed in parallel, except as noted in step 2.

Write Entries, HardState and Snapshot to persistent storage in order, i.e. Entries first, then HardState and Snapshot if they are not empty. If persistent storage supports atomic writes then all of them can be written together. Note that when writing an Entry with Index i, any previously-persisted entries with Index >= i must be discarded.
Send all Messages to the nodes named in the To field. It is important that no messages be sent until the latest HardState has been persisted to disk, and all Entries written by any previous Ready batch (Messages may be sent while entries from the same batch are being persisted). To reduce the I/O latency, an optimization can be applied to make leader write to disk in parallel with its followers (as explained at section 10.2.1 in Raft thesis). If any Message has type MsgSnap, call Node.ReportSnapshot() after it has been sent (these messages may be large). Note: Marshalling messages is not thread-safe; it is important to make sure that no new entries are persisted while marshalling. The easiest way to achieve this is to serialise the messages directly inside the main raft loop.
Apply Snapshot (if any) and CommittedEntries to the state machine. If any committed Entry has Type EntryConfChange, call Node.ApplyConfChange() to apply it to the node. The configuration change may be cancelled at this point by setting the NodeID field to zero before calling ApplyConfChange (but ApplyConfChange must be called one way or the other, and the decision to cancel must be based solely on the state machine and not external information such as the observed health of the node).
Call Node.Advance() to signal readiness for the next batch of updates. This may be done at any time after step 1, although all updates must be processed in the order they were returned by Ready.

Second, all persisted log entries must be made available via an implementation of the Storage interface. The provided MemoryStorage type can be used for this (if repopulating its state upon a restart), or a custom disk-backed implementation can be supplied.

Third, after receiving a message from another node, pass it to Node.Step:

func recvRaftRPC(ctx context.Context, m raftpb.Message) {
		n.Step(ctx, m)
	}

Finally, call Node.Tick() at regular intervals (probably via a time.Ticker). Raft has two important timeouts: heartbeat and the election timeout. However, internally to the raft package time is represented by an abstract "tick".

The total state machine handling loop will look something like this:

for {
    select {
    case <-s.Ticker:
      n.Tick()
    case rd := <-s.Node.Ready():
      saveToStorage(rd.State, rd.Entries, rd.Snapshot)
      send(rd.Messages)
      if !raft.IsEmptySnap(rd.Snapshot) {
        processSnapshot(rd.Snapshot)
      }
      for _, entry := range rd.CommittedEntries {
        process(entry)
        if entry.Type == raftpb.EntryConfChange {
          var cc raftpb.ConfChange
          cc.Unmarshal(entry.Data)
          s.Node.ApplyConfChange(cc)
        }
      }
      s.Node.Advance()
    case <-s.done:
      return
    }
  }

To propose changes to the state machine from the node to take application data, serialize it into a byte slice and call:

n.Propose(ctx, data)

If the proposal is committed, data will appear in committed entries with type raftpb.EntryNormal. There is no guarantee that a proposed command will be committed; the command may have to be reproposed after a timeout.

To add or remove node in a cluster, build ConfChange struct 'cc' and call:

n.ProposeConfChange(ctx, cc)

After config change is committed, some committed entry with type raftpb.EntryConfChange will be returned. This must be applied to node through:

var cc raftpb.ConfChange
	cc.Unmarshal(data)
	n.ApplyConfChange(cc)

Note: An ID represents a unique node in a cluster for all time. A
given ID MUST be used only once even if the old node has been removed.
This means that for example IP addresses make poor node IDs since they
may be reused. Node IDs must be non-zero.

Implementation notes

This implementation is up to date with the final Raft thesis (https://ramcloud.stanford.edu/~ongaro/thesis.pdf), although this implementation of the membership change protocol differs somewhat from that described in chapter 4. The key invariant that membership changes happen one node at a time is preserved, but in our implementation the membership change takes effect when its entry is applied, not when it is added to the log (so the entry is committed under the old membership instead of the new). This is equivalent in terms of safety, since the old and new configurations are guaranteed to overlap.

To ensure there is no attempt to commit two membership changes at once by matching log positions (which would be unsafe since they should have different quorum requirements), any proposed membership change is simply disallowed while any uncommitted change appears in the leader's log.

This approach introduces a problem when removing a member from a two-member cluster: If one of the members dies before the other one receives the commit of the confchange entry, then the member cannot be removed any more since the cluster cannot make progress. For this reason it is highly recommended to use three or more nodes in every cluster.

Overview (`hack/contrib/docker/grctl/README.md`)

Install

necessary condition

~/.kube/config exist

install shell

bash

docker run -it --rm -v /:/rootfs goodrain.me/rbd-grctl:V5.2-dev copy
grctl install

Overview (`hack/contrib/docker/registry-gc/README.md`)

Registry coordination image

This image contains the coordinator, the one-shot GC executor, and the native
Registry binary from the explicitly supplied image. Build it from the same Core
revision as rbd-api and publish it outside the Registry it will coordinate.

docker build \
  --build-arg REGISTRY_IMAGE="$REGISTRY_NATIVE_IMAGE" \
  -f hack/contrib/docker/registry-gc/Dockerfile \
  -t "$GC_COORDINATOR_IMAGE" .

REGISTRY_NATIVE_IMAGE must include the digest observed on the target cluster.
The build rejects mutable tags. Use /registry-coordinator as the sidecar command;
the same image supplies /registry-gc for server-generated GC Jobs.

Platform-scoped control authentication

Set these coordinator arguments when using the Console bridge:

text

--coordination-api=<console-origin>
--console-enterprise=<enterprise-id>
--console-region=<region-name>
--console-system-identity=true
--credential-file=/control/key
--permit-key-file=/permit/key

The control key is platform-owned and independent of the plugin gateway credential.
It must survive plugin uninstall/reinstall. The independent
permit key is shared only by rbd-api and the coordinator. Configure the API with
CLEANUP_REGISTRY_PERMIT_KEY_FILE=/var/run/cleanup-registry-permit/key and project
that Secret read-only into both components. Do not reuse the control key as the
permit key. No Region administrator token or private client certificate is
needed in signed Console mode. Plain HTTP requires explicit
--allow-internal-http=true for the trusted internal Console origin.

Console must include the system coordination endpoint and mount a platform-owned
Secret entry at the absolute path configured by CLEANUP_SYSTEM_COORDINATION_FILE.
That entry is one JSON document containing exactly enterprise, region, and
key (base64 encoding of the control key bytes). The enterprise and region must
match both the client scope and a current Console enterprise-region association.
The coordinator mounts the same control key bytes as /control/key; it does not
mount the JSON document. Use a generated printable key compatible with the CLI
credential-file parser. Never place values in application templates, logs or chat.
Missing or invalid configuration fails closed; there is no plugin-key fallback.
The full system URL is part of the signature, preventing replay to the plugin URL.

The default client mode remains the installation endpoint for existing callers.
Operator-managed Registry coordination explicitly enables system identity; GC
Jobs preserve that flag. Do not rotate a key underneath active GC without a
coordinated drain/restart: Console reloads the file, while clients hold the key
loaded at startup. Provisioning and controlled deactivation remain rollout gates.

Use separate Secrets and volumes for the two keys. GC Jobs inherit the Console
scope and control credential from the verified coordinator Pod, but never receive
the permit signing Secret. The launcher rejects shared Secret projections and
incomplete scopes. Invalid or missing explicit permit files fail closed, including
during rotation. Legacy direct-token mode remains supported when Console scope
arguments are absent.

Storage IDs, generations and physical volume identities must come from Core's
verified registry preparation response. Enrollment in collecting mode does not
authorize deletion. Storage, ingress, producer coverage, references, and original
execution identity still have to pass the existing readiness checks.

Deployment boundary

These binaries and authentication settings are prerequisites. Persistent Operator
support for the coordinator container and coordinated Service/probe configuration
must be installed before switching a managed rbd-hub. A manual Deployment edit
is not a durable installation method. Live ingress changes require the separate
review agreed for this rollout. GC remains an explicitly confirmed, one-shot job.

Initialize independent platform credentials

The installer helper creates only rbd-system/rainbond-registry-control and
rbd-system/rainbond-registry-permit. It does not patch workloads or ingress.
Select the target Kubernetes context explicitly:

python3 hack/contrib/docker/registry-gc/provision_identity.py \
  --context <verified-context> --enterprise <enterprise-id> --region <region-name>

Existing Secrets must have matching scope and purpose, no plugin ownerReference,
consistent Console projection, and independent keys. They are never overwritten.
A partial creation can be retried after checking state; a conflicting identity
requires investigation. The helper prints no credential data or kubectl errors.
Secrets are immutable to prevent accidental in-place rotation during active work.
Rotation requires separate versioned Secrets and a reviewed drain/migration.

The control Secret supplies key to the coordinator and console.json to Console.
The permit Secret supplies key only to Core API and the coordinator. Do not
project the entire control Secret into a plugin or permit Secret into GC Jobs.

Console runtime identity

The Console image must contain the private writer-announcement client. Configure
these non-sensitive downward-API entries on rbd-app-ui (RbdComponent spec.env):

yaml

- name: POD_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.name
- name: POD_UID
  valueFrom:
    fieldRef:
      fieldPath: metadata.uid

After a valid signed inventory request and enterprise/region ownership check,
Console announces its own Pod name/UID and the supported reference protocol using
its existing verified Region TLS client. Core observes the actual container and
image identity. No caller-provided image or writable flag is accepted. The
registration path is deliberately absent from the plugin/system bridge allowlist.
Missing identity or an older Core does not break read-only inventory, but supplies
no writer evidence. Success is cached per Console process and region; the record
is not a readiness grant. Completeness checks must still account for every live
Console instance; one successful request does not certify other replicas.

Overview (`third_party/coreos-etcd/contrib/raftexample/README.md`)

raftexample

raftexample is an example usage of etcd's raft library. It provides a simple REST API for a key-value store cluster backed by the Raft consensus algorithm.

Getting Started

Running single node raftexample

First start a single-member cluster of raftexample:

raftexample --id 1 --cluster http://127.0.0.1:12379 --port 12380

Each raftexample process maintains a single raft instance and a key-value server.
The process's list of comma separated peers (--cluster), its raft ID index into the peer list (--id), and http key-value server port (--port) are passed through the command line.

Next, store a value ("hello") to a key ("my-key"):

text

curl -L http://127.0.0.1:12380/my-key -XPUT -d hello

Finally, retrieve the stored key:

text

curl -L http://127.0.0.1:12380/my-key

Running a local cluster

First install goreman, which manages Procfile-based applications.

The Procfile script will set up a local example cluster. Start it with:

goreman start

This will bring up three raftexample instances.

Now it's possible to write a key-value pair to any member of the cluster and likewise retrieve it from any member.

Fault Tolerance

To test cluster recovery, first start a cluster and write a value "foo":

goreman start
curl -L http://127.0.0.1:12380/my-key -XPUT -d foo

Next, remove a node and replace the value with "bar" to check cluster availability:

goreman run stop raftexample2
curl -L http://127.0.0.1:12380/my-key -XPUT -d bar
curl -L http://127.0.0.1:32380/my-key

Finally, bring the node back up and verify it recovers with the updated value "bar":

goreman run start raftexample2
curl -L http://127.0.0.1:22380/my-key

Dynamic cluster reconfiguration

Nodes can be added to or removed from a running cluster using requests to the REST API.

For example, suppose we have a 3-node cluster that was started with the commands:

raftexample --id 1 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 --port 12380
raftexample --id 2 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 --port 22380
raftexample --id 3 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 --port 32380

A fourth node with ID 4 can be added by issuing a POST:

curl -L http://127.0.0.1:12380/4 -XPOST -d http://127.0.0.1:42379

Then the new node can be started as the others were, using the --join option:

raftexample --id 4 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379,http://127.0.0.1:42379 --port 42380 --join

The new node should join the cluster and be able to service key/value requests.

We can remove a node using a DELETE request:

curl -L http://127.0.0.1:12380/3 -XDELETE

Node 3 should shut itself down once the cluster has processed this request.

Design

The raftexample consists of three components: a raft-backed key-value store, a REST API server, and a raft consensus server based on etcd's raft implementation.

The raft-backed key-value store is a key-value map that holds all committed key-values.
The store bridges communication between the raft server and the REST server.
Key-value updates are issued through the store to the raft server.
The store updates its map once raft reports the updates are committed.

The REST server exposes the current raft consensus by accessing the raft-backed key-value store.
A GET command looks up a key in the store and returns the value, if any.
A key-value PUT command issues an update proposal to the store.

The raft server participates in consensus with its cluster peers.
When the REST server submits a proposal, the raft server transmits the proposal to its peers.
When raft reaches a consensus, the server publishes all committed updates over a commit channel.
For raftexample, this commit channel is consumed by the key-value store.

Overview (`third_party/coreos-etcd/hack/benchmark/README.md`)

Usage

Benchmark 3-member etcd cluster to get its read and write performance.

Instructions

Start 3-member etcd cluster on 3 machines
Update $leader and $servers in the script
Run the script in a separate machine

Caveat

Set environment variable GOMAXPROCS as the number of available cores to maximize CPU resources for both etcd member and bench process.
Set the number of open files per process as 10000 for amounts of client connections for both etcd member and benchmark process.

Overview (`third_party/coreos-etcd/hack/insta-discovery/README.md`)

Starts a cluster via the discovery service locally. Useful for testing.

Overview (`third_party/coreos-etcd/hack/kubernetes-deploy/README.md`)

etcd on Kubernetes

This is an example setting up etcd as a set of pods and services running on top of kubernetes. Using:

text

$ kubectl create -f etcd.yml 
services/etcd-client
pods/etcd0
services/etcd0
pods/etcd1
services/etcd1
pods/etcd2
services/etcd2
$ # now deploy a service that consumes etcd, such as vulcand
$ kubectl create -f vulcand.yml

TODO:

create a replication controller like service that knows how to add and remove nodes from the cluster correctly
use kubernetes secrets API to configure TLS for etcd clients and peers

Overview (`third_party/coreos-etcd/hack/patch/README.md`)

./hack/patch/cherrypick.sh

Handles cherry-picks of PR(s) from etcd master to a stable etcd release branch automatically.

Setup

Set the UPSTREAM_REMOTE and FORK_REMOTE environment variables.
UPSTREAM_REMOTE should be set to git remote name of github.com/coreos/etcd,
and FORK_REMOTE should be set to the git remote name of the forked etcd
repo (github.com/${github-username}/etcd). Use git remotes -v to
look up the git remote names. If etcd has not been forked, create
one on github.com and register it locally with git remote add ....

text

export UPSTREAM_REMOTE=origin
export FORK_REMOTE=${github-username}
export GITHUB_USER=${github-username}

Next, install hub from https://github.com/github/hub

Usage

To cherry pick PR 12345 onto release-3.2 and propose is as a PR, run:

./hack/patch/cherrypick.sh ${UPSTREAM_REMOTE}/release-3.2 12345

To cherry pick 12345 then 56789 and propose them togther as a single PR, run:

text

./hack/patch/cherrypick.sh ${UPSTREAM_REMOTE}/release-3.2 12345 56789

Overview (`third_party/coreos-etcd/hack/tls-setup/README.md`)

This demonstrates using Cloudflare's cfssl to easily generate certificates for an etcd cluster.

Defaults generate an ECDSA-384 root and leaf certificates for localhost. etcd nodes will use the same certificates for both sides of mutual authentication, but won't require client certs for non-peer clients.

Instructions

Install git, go, and make
Amend https://github.com/coreos/etcd/blob/master/hack/tls-setup/config/req-csr.json - IP's currently in the config should be replaced/added with IP addresses of each cluster node, please note 127.0.0.1 is always required for loopback purposes:

json

Example:
{
  "CN": "etcd",
  "hosts": [
    "3.8.121.201",
    "46.4.19.20",
    "127.0.0.1"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 384
  },
  "names": [
    {
      "O": "autogenerated",
      "OU": "etcd cluster",
      "L": "the internet"
    }
  ]
}

Run make to generate the certs

Overview (`third_party/coreos-etcd/contrib/systemd/etcd2-backup-coreos/README.md`)

etcd2-backup-coreos

Remote backup and multi-node restore services for etcd2 clusters on CoreOS Linux.

Warning: This package is only intended for use on CoreOS Linux.

Terminology

Founding member : The node which is the first member of the new recovered cluster. It is this node's rclone backup data (only) that will be used to restore the cluster. The rest of the nodes will join the cluster with no data, and simply catch up with the founding member.

Configuration

Before installing etcd2-backup, configure 30-etcd2-backup-restore.conf:

text

[Service]
Environment="ETCD_RESTORE_MASTER_ADV_PEER_URLS=<http://host:port>"
Environment="RCLONE_ENDPOINT=remote-name:path/to/backups"

Assuming a deployment to CoreOS with etcd2, only change:

ETCD_RESTORE_MASTER_ADV_PEER_URLS
This is the new advertised peer url of the new etcd2 node that will be the founding member of the new restored cluster. We will call this node the founding member.
RCLONE_ENDPOINT
The rclone endpoint to which backups will be stored.

Feel free to point any number of machines at the same RCLONE_ENDPOINT, path and all. Backups for each machine are stored in a sub-folder named with the machine ID (%m in systemd parlance)
./rclone.conf
The rclone configuration file which will be installed. Must list a [section] which matches RCLONE_ENDPOINT's remote-name component.

An easy way to generate this config file is to install rclone on a local machine. Then follow the configuration instructions to generate an rclone.conf file.

To adjust backup frequency, edit ./etcd2-backup.timer

Installation

Once those things are configured, run ./build.

The build script generates a tarball for copying to CoreOS instances. The tarball contains the etcd2-backup-install script.

After extracting the contents of the tar file and running the install script, three new systemd services are added. One service, etcd2-backup, performs periodic etcd backups, while the other two services, etcd2-restore and etcd2-join, handle restore procedures.

etcd2-backup.service
A oneshot service which calls etcdctl backup and syncs the backups to the rclone endpoint (using an rclone container, of course). etcd2-backup.timer is responsible for periodically running this service.
etcd2-restore.service
A oneshot service which wipes all etcd2 data and restores a single-node cluster from the rclone backup. This is for restoring the founding member only.
etcd2-join.service
A oneshot service which wipes all etcd2 data and re-joins the new cluster. This is for adding members after the founding member has succesfully established the new cluster via etcd2-restore.service

Recovery

This assumes that the cluster has lost quorum and is not recoverable. Otherwise try to heal the cluster first.

Backup Freshness

Two factors contribute to the relative freshness or staleness of a backup. The etcd2-backup.timer takes a backup every 30 seconds by default, and the etcd snapshot-count option controls how many transactions are committed between each write of the snapshot to permanent storage. Given those parameters, we can compute the upper bound on the outdatedness of a backup.
Assumptions:

transaction rate is a constant 1000 transactions / second
etcd2-backup.timer is configured for a 30 second interval
etcd2 snapshot-count=10000

text

max-missed-seconds= (10000 transactions / (1000 transactions / second)) + 30 seconds = 40 seconds

Recovery Procedure

Make sure etcd2.service and etcd2-backup.timer are stopped on all nodes in the cluster
Restore the founding member by starting etcd2-restore.service and then, if successful, etcd2.service
Restore the rest of the cluster one at a time. Start etcd2-join.service, and then, if successful, etcd2.service. Please verify with etcdctl cluster-health that the expected set of nodes is present and healthy after each node joins.
Verify that the data is sane (enough). If so, kick off etcd2-backup.timer on all nodes and, hopefully, go back to bed.

Retroactively change the founding member

It is necessary to change the cluster's founding member in order to restore a cluster from any other node's data.

Change the value of ETCD_RESTORE_MASTER_ADV_PEER_URLS in 30-etcd2-backup-restore.conf to the advertised peer url of the new founding member. Repeat the install process above on all nodes in the cluster, then proceed with the recovery procedure.

Example

Let's pretend that we have an initial 3 node CoreOS cluster that we want to back up to S3.

ETCD_NAME	ETCD_ADVERTISED_PEER_URL
e1	http://172.17.4.51:2379
e2	http://172.17.4.52:2379
e3	http://172.17.4.53:2379

In the event that the cluster fails, we want to restore from e1's backup

Configuration

text

[Service]
Environment="ETCD_RESTORE_MASTER_ADV_PEER_URLS=http://172.17.4.51:2379"
Environment="RCLONE_ENDPOINT=s3-testing-conf:s3://etcd2-backup-bucket/backups"

The ./rclone.conf file must contain a [section] matching RCLONE_ENDPOINTS's remote-name component.

text

[s3-testing-conf]
type = s3
access_key_id = xxxxxxxx
secret_access_key = xxxxxx
region = us-west-1
endpoint =
location_constraint =

Installation

cd etcd2-backup
./build
scp etcd2-backup.tgz core@e1:~/
ssh core@e1
e1 $  mkdir -p ~/etcd2-backup
e1 $  mv etcd2-backup.tgz etcd2-backup/
e1 $ cd etcd2-backup
e1 $ tar zxvf ~/etcd2-backup.tgz
e1 $ ./etcd2-backup-install
# Only do the following two commands if this node should generate backups
e1 $ sudo systemctl enable etcd2-backup.timer
e1 $ sudo systemctl start etcd2-backup.timer

e1 $ exit

Now e1's etcd data will be backed up to s3://etcd2-backup-bucket/backups/<e1-machine-id>/ according to the schedule described in etcd2-backup.timer.

Repeat the process for e2 and e3. To stop a node from generating backups, omit enabling and starting etcd2-backup.timer.

Restore the cluster

Let's assume that a mischievous friend decided it would be a good idea to corrupt the etcd2 data-dir on ALL of the nodes (e1,e2,e3). Simply restore the cluster from e1's backup.

Here's how to recover:

# First, ENSURE etcd2 and etcd2-backup are not running on any nodes
for node in e{1..3};do
    ssh core@$node "sudo systemctl stop etcd2.service etcd2-backup.{timer,service}"
done

ssh core@e1 "sudo systemctl start etcd2-restore.service && sudo systemctl start etcd2.service"

for node in e{2..3};do
    ssh core@$node "sudo systemctl start etcd2-join.service && sudo systemctl start etcd2.service"
    sleep 10
done

After e2 and e3 finish catching up, the cluster should be back to normal.

Migrate the cluster

The same friend who corrupted the etcd2 data-dirs decided that to have more fun. This time, the friend dumps coffee on the machines hosting e1, e2 and e3. There is a horrible smell, and the machines are dead.

Luckily, there's a new 3-node etcd2 cluster ready to go, along with the S3 backup for e1 from the old cluster.

The new cluster configuration looks like this. Assume that etcd2-backup is not installed. (If it is, make sure it's not running on any nodes)

ETCD_NAME	ETCD_ADVERTISED_PEER_URL
q1	http://172.17.8.201:2379
q2	http://172.17.8.202:2379
q3	http://172.17.8.203:2379

We will assume q1 is the chosen founding member, though picking any node is fine.

Migrate the remote backup

First, copy the backup from e1's backup folder to q1's backup folder. I will show the S3 example.

# Make sure to remove q1's backup directory, if it exists already
aws s3 rm --recursive s3://etcd2-backup-bucket/backups/<q1-machine-id>
aws s3 cp --recursive s3://etcd2-backup-bucket/backups/<e1-machine-id> s3://etcd2-backup-bucket/backups/<q1-machine-id>

Configure the New Cluster

text

[Service]
Environment="ETCD_RESTORE_MASTER_ADV_PEER_URLS=http://172.17.8.201:2379"
Environment="RCLONE_ENDPOINT=s3-testing-conf:s3://etcd2-backup-bucket/backups"

Since this is a new cluster, each new node will have a new machine-id and will not clobber the backups from the old cluster, even though RCLONE_ENDPOINT is the same for both the old e cluster and the new q cluster.

Installation

We first want to install the configured etcd2-backup package on all nodes, but not start any services yet.

cd etcd2-backup
./build
for node in q{1..3};do
    scp etcd2-backup.tgz core@$node:~/
    ssh core@$node "mkdir -p ~/etcd2-backup"
    ssh core@$node "mv etcd2-backup.tgz etcd2-backup/"
    ssh core@$node " cd etcd2-backup"
    ssh core@$node " tar zxvf ~/etcd2-backup.tgz"
    ssh core@$node " ./etcd2-backup-install"
done

Migrate the Cluster

With q1 as the founding member.

# First, make SURE etcd2 and etcd2-backup are not running on any nodes

for node in q{1..3};do
    ssh core@$node "sudo systemctl stop etcd2.service"
done

ssh core@q1 "sudo systemctl start etcd2-restore.service && sudo systemctl start etcd2.service"

for node in q{2..3};do
    ssh core@$node "sudo systemctl start etcd2-join.service && sudo systemctl start etcd2.service"
    sleep 10
done

After confirming the cluster has migrated properly, start and enable etcd2-backup.timer on at least one node.

ssh core@q1 "sudo systemctl enable etcd2-backup.service && sudo systemctl start etcd2-backup.service"

There should now be periodic backups going to: s3://etcd2-backup-bucket/backups/<q1-machine-id>

Words of caution

Notice the sleep 10 commands that follow starting etcd2-join.service and then etcd2.service. This sleep is there to allow the member that joined to cluster time to catch up on the cluster state before we attempt to add the next member. This involves sending the entire snapshot over the network. If the dataset is large, the network between nodes is slow, disks are already bogged down, or the system is otherwise overutilized, try increasing the sleep.

In the case of large data sets, it is recommended to copy the data directory produced by etcd2-restore on the founding member to the other nodes before running etcd2-join on them. This will avoid etcd transferring the entire snapshot to every node after it joins the cluster.
It is not recommended clients be allowed to access the etcd2 cluster until all members have been added and finished catching up.

Overview (`third_party/coreos-etcd/contrib/systemd/etcd3-multinode/README.md`)

etcd3 multi-node cluster

Here's how to deploy etcd cluster with systemd.

Set up data directory

etcd needs data directory on host machine. Configure the data directory accessible to systemd as:

text

sudo mkdir -p /var/lib/etcd
sudo chown -R root:$(whoami) /var/lib/etcd
sudo chmod -R a+rw /var/lib/etcd

Write systemd service file

In each machine, write etcd systemd service files:

text

cat > /tmp/my-etcd-1.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=etcd --name my-etcd-1 \
    --data-dir /var/lib/etcd \
    --listen-client-urls http://${IP_1}:2379 \
    --advertise-client-urls http://${IP_1}:2379 \
    --listen-peer-urls http://${IP_1}:2380 \
    --initial-advertise-peer-urls http://${IP_1}:2380 \
    --initial-cluster my-etcd-1=http://${IP_1}:2380,my-etcd-2=http://${IP_2}:2380,my-etcd-3=http://${IP_3}:2380 \
    --initial-cluster-token my-etcd-token \
    --initial-cluster-state new

[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/my-etcd-1.service /etc/systemd/system/my-etcd-1.service

text

cat > /tmp/my-etcd-2.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=etcd --name my-etcd-2 \
    --data-dir /var/lib/etcd \
    --listen-client-urls http://${IP_2}:2379 \
    --advertise-client-urls http://${IP_2}:2379 \
    --listen-peer-urls http://${IP_2}:2380 \
    --initial-advertise-peer-urls http://${IP_2}:2380 \
    --initial-cluster my-etcd-1=http://${IP_1}:2380,my-etcd-2=http://${IP_2}:2380,my-etcd-3=http://${IP_3}:2380 \
    --initial-cluster-token my-etcd-token \
    --initial-cluster-state new

[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/my-etcd-2.service /etc/systemd/system/my-etcd-2.service

text

cat > /tmp/my-etcd-3.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=etcd --name my-etcd-3 \
    --data-dir /var/lib/etcd \
    --listen-client-urls http://${IP_3}:2379 \
    --advertise-client-urls http://${IP_3}:2379 \
    --listen-peer-urls http://${IP_3}:2380 \
    --initial-advertise-peer-urls http://${IP_3}:2380 \
    --initial-cluster my-etcd-1=http://${IP_1}:2380,my-etcd-2=http://${IP_2}:2380,my-etcd-3=http://${IP_3}:2380 \
    --initial-cluster-token my-etcd-token \
    --initial-cluster-state new

[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/my-etcd-3.service /etc/systemd/system/my-etcd-3.service

Start the service

The service needs to be enabled first, in case of system reboot:

text

sudo systemctl daemon-reload
sudo systemctl enable my-etcd-1.service
sudo systemctl start my-etcd-1.service

text

sudo systemctl daemon-reload
sudo systemctl enable my-etcd-2.service
sudo systemctl start my-etcd-2.service

text

sudo systemctl daemon-reload
sudo systemctl enable my-etcd-3.service
sudo systemctl start my-etcd-3.service

Check logs

systemd stores etcd server logs with journald:

text

sudo systemctl status my-etcd-1.service -l --no-pager
sudo journalctl -u my-etcd-1.service -l --no-pager|less
sudo journalctl -f -u my-etcd-1.service

text

sudo systemctl status my-etcd-2.service -l --no-pager
sudo journalctl -u my-etcd-2.service -l --no-pager|less
sudo journalctl -f -u my-etcd-2.service

text

sudo systemctl status my-etcd-3.service -l --no-pager
sudo journalctl -u my-etcd-3.service -l --no-pager|less
sudo journalctl -f -u my-etcd-3.service

Stop etcd

To disable etcd process:

text

sudo systemctl stop my-etcd-1.service
sudo systemctl disable my-etcd-1.service

text

sudo systemctl stop my-etcd-2.service
sudo systemctl disable my-etcd-2.service

text

sudo systemctl stop my-etcd-3.service
sudo systemctl disable my-etcd-3.service

Plans 2026 07 22 Cmd Args Yaml Design (`docs/plans/2026-07-22-cmd-args-yaml-design.md`)

组件视图 Kubernetes 属性 cmd/args YAML 化设计文档

一、项目背景

1.1 项目架构

Rainbond 组件 Kubernetes 属性由三层协作完成：

text

rainbond-ui 组件视图 Kubernetes 属性页
    -> rainbond-console /console/teams/{team}/components/{component}/k8s-attributes
    -> rainbond /v2/tenants/{tenant}/services/{service}/k8s-attributes
    -> worker 转换为 Kubernetes Container.Command / Container.Args

1.2 现有基础

rainbond 已有 component_k8s_attributes 通用表，字段包含 name、save_type、attribute_value。
rainbond 已定义 cmd 和 args 属性名，并在 worker/appm/conversion/version.go 中应用到容器 Command 和 Args。
rainbond-console 通过 console.services.k8s_attribute 保存 console 侧记录并同步到 region。
rainbond-ui 的组件 Kubernetes 属性页已支持 JSON、YAML、string、boolean 等编辑模式。

1.3 核心需求

当前 cmd 被 UI 当作字符串输入，region worker 在解析失败时使用 strings.Split(value, " ") 兜底，导致包含空格或 shell 结构的一行命令被错误拆分，例如：

text

/entrypoint.sh;while true;do echo hello >/dev/null;sleep 1;done

目标是支持 cmd 和 args 以 YAML 数组格式录入、存储和解析，避免任何空格切割。

二、用户旅程（MUST — 禁止跳过）

2.1 用户操作流程

用户如何配置/触发该功能？（必须回答，若需要 UI 则列出页面）
用户进入组件详情 -> Kubernetes 属性视图，点击添加属性，选择 cmd 或 args，在 YAML 编辑器中录入数组。
用户如何查看状态/结果？
Kubernetes 属性列表中 cmd 和 args 按 YAML 类型展示，用户点击编辑可看到原 YAML 内容。
管理员/审批人如何操作？
不涉及额外管理员或审批流程，沿用现有组件配置权限。

2.2 页面原型

页面：组件详情 -> Kubernetes 属性视图。
弹窗/抽屉：添加/编辑 Kubernetes 属性抽屉。
表单交互：
- cmd 示例：
  yaml
```
- /entrypoint.sh
```
- args 示例：
  yaml
```
- /entrypoint.sh;while true;do echo hello >/dev/null;sleep 1;done
```
- 提示文案说明每一行 - ... 是 Kubernetes argv 的一个元素，不会按空格切割。

2.3 外部系统交互

不涉及 webhook、回调、通知或第三方集成。最终变更体现在 Kubernetes Pod template 的 containers[].command 和 containers[].args 字段。

三、整体架构设计

3.1 系统架构图

text

UI YAML editor
  saves { name: "cmd"|"args", save_type: "yaml", attribute_value: "- ..." }
      |
Console k8s_attribute service
  stores YAML string and forwards unchanged
      |
Rainbond region API
  stores save_type/attribute_value
      |
Worker conversion
  parses YAML/JSON sequence into []string
      |
Kubernetes Container.Command / Container.Args

3.2 核心流程

UI 将 cmd 和 args 纳入 YAML 字段列表，并从 string 字段中移除 cmd。
UI 添加 args 到可选 Kubernetes 属性列表，并更新 cmd/args 提示示例。
console 对 save_type=yaml 保持原文存储和透传，不做空格拆分。
region worker 对 cmd/args 只解析 YAML/JSON sequence；旧 save_type=string 或无法解析的历史值整体作为单个 argv 元素。
compose 检测等内部来源继续可用，但优先保存为 YAML sequence，保证与 UI 契约一致。

四、数据模型设计

4.1 新增数据库表

不新增数据库表。

4.2 数据关系

继续使用 component_k8s_attributes：

text

tenant_id
component_id
name            cmd / args
save_type       yaml
attribute_value YAML sequence string

旧数据兼容规则：

JSON 数组旧值继续按数组解析。
YAML 数组新值按数组解析。
无法解析为数组的历史字符串不再空格切割，整体作为一个参数。

五、API设计

5.1 接口列表

沿用现有接口，不新增路由：

GET /console/teams/{team_name}/components/{service_alias}/k8s-attributes
POST /console/teams/{team_name}/components/{service_alias}/k8s-attributes
PUT /console/teams/{team_name}/components/{service_alias}/k8s-attributes/{name}
region 对应 /v2/tenants/{tenant_name}/services/{service_alias}/k8s-attributes

5.2 请求/响应结构

新增推荐请求格式：

json

{
  "attribute": {
    "name": "args",
    "save_type": "yaml",
    "attribute_value": "- /entrypoint.sh;while true;do echo hello >/dev/null;sleep 1;done\n"
  }
}

cmd 同样使用 save_type=yaml，值必须表达字符串数组。

六、核心实现设计

6.1 关键逻辑

rainbond：
- 新增或提取 parseStringSequenceAttribute 类辅助逻辑，按 YAML/JSON 数组解析 cmd/args。
- 当解析结果不是字符串数组时返回错误，避免静默生成错误 Pod spec。
- 对旧字符串值整体保留为 []string{value}，不再使用 strings.Split(value, " ")。
rainbond-console：
- app_check_service 中自动保存 compose command/args 时改为 YAML sequence，save_type=yaml。
- k8s_attribute_service 对用户提交的 cmd/args 做轻量校验：必须是 YAML/JSON 数组，且每项是字符串。
rainbond-ui：
- DEFAULT_ATTRIBUTE_FIELDS 增加 args。
- YAML_FIELDS 增加 cmd、args；STRING_FIELDS 移除 cmd。
- 更新 TooltipValueArr 和输入提示文案，突出“不按空格切割”。

6.2 复用现有代码

复用现有 CodeMirrorForm YAML 编辑器。
复用现有 save_type=yaml 存储和展示逻辑。
复用现有 console -> region API 透传链路。

七、实施计划

跨层覆盖检查（MUST）

Go (rainbond): 需要 — worker 解析 cmd/args YAML/JSON sequence，禁止空格拆分 fallback，补 Go 测试。
Python (console): 需要 — 校验并保存 cmd/args YAML sequence，内部检测来源也改为 YAML 存储，补 Python 测试。
React (rainbond-ui): 需要 — 组件 Kubernetes 属性页增加 args，把 cmd/args 切为 YAML 编辑并更新提示。
Plugin frontend (enterprise-base): 不涉及 — 该功能在主平台组件视图。
Plugin backend (plugin-template): 不涉及 — 无插件 API 变更。

Sprint 1: Region 解析契约

Task 1.1: 补充 cmd/args 解析测试

仓库：rainbond
文件：worker/appm/conversion/version_test.go 或新增聚焦测试文件
实现内容：覆盖 YAML 数组、JSON 数组、单个 shell 字符串 fallback。
验收标准：旧空格切割场景失败后修复通过。

Task 1.2: 实现 YAML/JSON sequence 解析

仓库：rainbond
文件：worker/appm/conversion/version.go
实现内容：替换 strings.Split fallback，解析失败时整体作为单元素。
验收标准：go test ./worker/appm/conversion -run CmdArgs -v 通过。

Sprint 2: Console 保存与校验

Task 2.1: 补充 k8s attribute service 测试

仓库：rainbond-console
文件：console/tests/k8s_attribute_service_test.py
实现内容：验证 cmd/args YAML 数组保存透传，非数组报错。
验收标准：相关 pytest 通过。

Task 2.2: 实现 console 校验和 compose 保存格式

仓库：rainbond-console
文件：console/services/k8s_attribute.py、console/services/app_check_service.py
实现内容：cmd/args 入库前规范为 YAML sequence，内部检测保存使用 save_type=yaml。
验收标准：相关 pytest 通过。

Sprint 3: UI 编辑体验

Task 3.1: 修改 Kubernetes 属性页面

仓库：rainbond-ui
文件：src/pages/Component/kubernets.js
实现内容：cmd/args 使用 YAML 编辑器，新增示例和提示。
验收标准：yarn build 通过。

Task 3.2: 更新 i18n 文案

仓库：rainbond-ui
文件：src/locales/zh-CN/*.js、src/locales/en-US/*.js
实现内容：更新 cmd/args 提示文案。
验收标准：页面引用的文案 key 完整。

八、关键参考代码

功能	文件	说明
region API 路由	`api/api_routers/version2/v2Routers.go:610`	`/k8s-attributes` 路由
region 属性模型	`api/model/component.go:276`	`ComponentK8sAttribute`
worker 容器转换	`worker/appm/conversion/version.go:480`	`cmd/args` 应用到 Kubernetes Container
console 服务	`console/services/k8s_attribute.py:15`	console 保存并同步 region
console 视图	`console/views/k8s_attribute.py:14`	console k8s attribute API
UI 属性页面	`src/pages/Component/kubernets.js:25`	Kubernetes 属性可选字段和编辑器选择

Third Party Coreos Etcd Etcdctl Doc Mirror Maker (`third_party/coreos-etcd/etcdctl/doc/mirror_maker.md`)

Mirror Maker

Mirror maker mirrors a prefix in the key-value space of an etcd cluster into another prefix in another cluster. Mirroring is designed for copying configuration to various clusters distributed around the world. Mirroring usually has very low latency once it completes synchronizing with the initial state. Mirror maker utilizes the etcd watcher facility to immediately inform the mirror of any key modifications. Based on our experiments, the network latency between the mirror maker and the two clusters accounts for most of the latency. If the network is healthy, copying configuration held in etcd to the mirror should take under one second even for a world-wide deployment.

If the mirror maker fails to connect to one of the clusters, the mirroring will pause. Mirroring can be resumed automatically once connectivity is reestablished.

The mirroring mechanism is unidirectional. Data under the destination cluster’s mirroring prefix should be treated as read only. The mirror maker only mirrors key-value pairs; metadata, such as version number or modification revision, is discarded. However, mirror maker still attempts to preserve update ordering during normal operation, but there is no ordering guarantee during initial sync nor during failure recovery following network interruption. As a rule of thumb, the ordering of the updates on the mirror should not be considered reliable.

text

+-------------+
|             |
|  source     |      +-----------+
|  cluster    +----> |  mirror   |
|             |      |  maker    |
+-------------+      +---+-------+
                         |
                         v
               +-------------+
               |             |
               |    mirror  |
               |    cluster  |
               |             |
               +-------------+

Mirror-maker is a built-in feature of etcdctl.

--- METRICS ---

Files Extracted: 28
Estimated Token Budget: ~34141 tokens
Canonical Reference: https://codewiki.google/github.com/goodrain/rainbond

Docker & Self-Hosted Deployment Verified OCI

Technical Documentation: goodrain/rainbond

Overview (`README.md`) Copy Section

What is Rainbond? Copy Chapter

Where to start Copy Chapter

What problems Rainbond solves Copy Chapter

1. Deliver applications without deeply learning Kubernetes Copy Section

2. Make delivery in complex enterprise environments more reliable Copy Section

3. Standardize delivery with marketplace and templates Copy Section

4. Keep operations and delivery centered around the application Copy Section

Why Rainbond Copy Chapter

Lower barrier, not lower capability Copy Section

Better fit for complex enterprise environments Copy Section

Stronger marketplace and standardized delivery capability Copy Section

How it differs from common platform choices Copy Chapter

Typical scenarios Copy Chapter

Quick start Copy Chapter

Requirements Copy Section

Quick install Copy Section

Next step Copy Section

Community and support Copy Chapter

Contributing Copy Chapter

Related projects Copy Chapter

License Copy Chapter

Overview (`LICENSES/README.md`) Copy Section

Third-Party Licenses

Directory Structure Copy Chapter

Purpose Copy Chapter

Overview (`plugins/README.md`) Copy Section

Rainbond Plugins Copy Chapter

License Copy Chapter

Overview (`third_party/coreos-etcd/README.md`) Copy Section

etcd

Community meetings Copy Chapter

Getting started Copy Chapter

Getting etcd Copy Section

Running etcd Copy Section

etcd TCP ports Copy Section

Running a local etcd cluster Copy Section

Running etcd on Kubernetes Copy Section

Next steps Copy Section

Contact Copy Chapter

Contributing Copy Chapter

Reporting bugs Copy Chapter

License Copy Section

Overview (`third_party/process-exporter/README.md`) Copy Section

process-exporter

Installation Copy Chapter

Running Copy Chapter

Configuration and group naming Copy Chapter

Using a config file Copy Section

Using a config file: group name

Using a config file: process selectors

Using -procnames/-namemapping instead of config.path Copy Section

Group Metrics Copy Chapter

num_procs gauge Copy Section

cpu_seconds_total counter Copy Section

read_bytes_total counter Copy Section

write_bytes_total counter Copy Section

major_page_faults_total counter Copy Section

minor_page_faults_total counter Copy Section

context_switches_total counter Copy Section

memory_bytes gauge Copy Section

open_filedesc gauge Copy Section

worst_fd_ratio gauge Copy Section

oldest_start_time_seconds gauge Copy Section

num_threads gauge Copy Section

states gauge Copy Section

Group Thread Metrics Copy Chapter

thread_count gauge Copy Section

thread_cpu_seconds_total counter Copy Section

thread_io_bytes_total counter Copy Section

thread_major_page_faults_total counter Copy Section

thread_minor_page_faults_total counter Copy Section

thread_context_switches_total counter Copy Section

Instrumentation cost Copy Chapter

Dashboards Copy Chapter

Building Copy Chapter

Overview (`util/http/README.md`) Copy Section

用法实例 Copy Section

Overview (`README.md`)

What is Rainbond?

Where to start

What problems Rainbond solves

1. Deliver applications without deeply learning Kubernetes

2. Make delivery in complex enterprise environments more reliable

3. Standardize delivery with marketplace and templates

4. Keep operations and delivery centered around the application

Why Rainbond

Lower barrier, not lower capability

Better fit for complex enterprise environments

Stronger marketplace and standardized delivery capability

How it differs from common platform choices

Typical scenarios

Quick start

Requirements

Quick install

Next step

Community and support

Contributing

Related projects

License

Overview (`LICENSES/README.md`)

Directory Structure

Purpose

Overview (`plugins/README.md`)

Rainbond Plugins

License

Overview (`third_party/coreos-etcd/README.md`)

Community meetings

Getting started

Getting etcd

Running etcd

etcd TCP ports

Running a local etcd cluster

Running etcd on Kubernetes

Next steps

Contact

Contributing

Reporting bugs

License

Overview (`third_party/process-exporter/README.md`)

Installation

Running

Configuration and group naming

Using a config file

Using -procnames/-namemapping instead of config.path

Group Metrics

num_procs gauge

cpu_seconds_total counter

read_bytes_total counter

write_bytes_total counter

major_page_faults_total counter

minor_page_faults_total counter

context_switches_total counter

memory_bytes gauge

open_filedesc gauge

worst_fd_ratio gauge

oldest_start_time_seconds gauge

num_threads gauge

states gauge

Group Thread Metrics

thread_count gauge

thread_cpu_seconds_total counter

thread_io_bytes_total counter

thread_major_page_faults_total counter

thread_minor_page_faults_total counter

thread_context_switches_total counter

Instrumentation cost

Dashboards

Building

Overview (`util/http/README.md`)

用法实例

Validation Rules

Overview (`util/termtables/README.md`)

Installation

Go Style Documentation

APC Command Line usage

Basic Usage

Advanced Usage