# Technical Documentation: goodrain/rainbond > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/goodrain/rainbond) ### Overview (`README.md`)
Rainbond Logo

中文

GitHub stars Rainbond version Discord

An open-source container platform. No Kubernetes expertise required.

Build, deploy, assemble, and manage applications on Kubernetes through a graphical interface, without K8s expertise.

Website · Documentation

## What is Rainbond? Rainbond is an open-source container platform that requires no Kubernetes expertise. Its core capabilities are 100% open source. It abstracts away infrastructure complexity and provides a unified way to deploy and manage business applications, AI-generated projects, open-source AI software, and large language model services. AI helps teams deploy and operate these workloads, keeping applications running reliably on their own servers or Kubernetes clusters. ## Where to start | Your goal | Start here | | --- | --- | | I use AI coding and want to deploy my project | [Install RainSkills](https://github.com/goodrain/rainskills) | | I want to try Rainbond without preparing a server | [Use Rainbond Cloud](https://run.rainbond.com) | | I want to run applications on my own servers or Kubernetes | [Install Rainbond privately](https://www.rainbond.com/docs/quick-start/quick-install) | | I want to deploy open-source applications such as Dify or RAGFlow | [Visit the Rainbond Application Marketplace](https://hub.rainbond.com) | | I am evaluating open-source container platforms | [Explore Rainbond's application management and delivery capabilities](https://www.rainbond.com/compare) | --- ## What problems Rainbond solves ### 1. Deliver applications without deeply learning Kubernetes Rainbond supports deploying applications from source code, container images, Docker Compose, Helm, or application templates. Its graphical interface and standardized workflows bring application dependencies, access, upgrades, and rollbacks into one place. ### 2. Make delivery in complex enterprise environments more reliable Rainbond is especially suitable for: - Private deployment - Internal-network deployment - Offline delivery - Xinchuang adaptation - x86 to ARM migration validation ### 3. Standardize delivery with marketplace and templates Rainbond is not only about “getting apps running”. It also provides: - Application templates - Application marketplace - One-click installation - One-click upgrade - Customer environment replication ### 4. Keep operations and delivery centered around the application Rainbond emphasizes: - Application-level abstraction - Application topology and dependencies - Application upgrades and rollbacks - Delivery and operation across multiple environments and clusters --- ## Why Rainbond ### Lower barrier, not lower capability Rainbond does not simply hide Kubernetes. It redistributes complexity: - The platform and operations teams absorb low-level concerns - Development and delivery teams focus on building, deploying, delivering, and operating applications ### Better fit for complex enterprise environments Many platforms are strong at cluster management. Rainbond is stronger when the real problem is: - How to deliver applications - How to replicate delivery to customer environments - How to handle upgrades in offline environments - How to privately deploy AI applications ### Stronger marketplace and standardized delivery capability If what you really need is: - Template-based delivery - Marketplace reuse - Offline package export/import - Versioned upgrades and rollbacks Rainbond is closer to the actual work. --- ## How it differs from common platform choices | Dimension | Rainbond | Rancher / KubeSphere type platforms | | --- | --- | --- | | Primary focus | Application delivery, application management, template reuse | Cluster governance, platform ops, Kubernetes management | | Typical users | Developers, delivery teams, enterprise IT, platform owners | K8s admins, platform ops, cluster governance teams | | Strongest scenarios | Private deployment, offline, Xinchuang, marketplace delivery, AI privatization | Multi-cluster governance, resource management, platform ops | | Learning curve | Lower, exposes less K8s detail | Requires stronger K8s and platform governance knowledge | If you are in evaluation mode, start here: - [Rainbond vs KubeSphere](https://www.rainbond.com/compare/rainbond-vs-kubesphere?channel=github) - [Rainbond vs Rancher](https://www.rainbond.com/compare/rainbond-vs-rancher?channel=github) - [Rainbond vs Sealos](https://www.rainbond.com/compare/rainbond-vs-sealos?channel=github) --- ## Typical scenarios - **Deliver applications without deeply learning Kubernetes** - **Privately deploy AI applications** - **Offline and internal-network delivery** - **Xinchuang and ARM migration** - **Enterprise marketplace and standardized delivery** Continue with: - [Offline / Xinchuang Topic](https://www.rainbond.com/offline-and-xinchuang?channel=github) - [Comparison Center](https://www.rainbond.com/compare?channel=github) --- ## Quick start ### Requirements - Linux or macOS - Recommended: at least 2 CPU / 8GB RAM / 50GB disk ### Quick install Run: ```bash curl -o install.sh https://get.rainbond.com && bash ./install.sh ``` Then open: ```bash http://:7070 ``` ### Next step 1. [Quick Install](https://www.rainbond.com/docs/quick-start/quick-install?channel=github) 2. [Deploy your first app](https://www.rainbond.com/docs/quick-start/getting-started?channel=github) 3. [Explore the Marketplace](https://hub.rainbond.com?channel=github) --- ## Community and support - [Documentation](https://www.rainbond.com/docs?channel=github) - [FAQ](https://www.rainbond.com/docs/faq?channel=github) - [Community Support](https://www.rainbond.com/docs/support?channel=github) - [GitHub Issues](https://github.com/goodrain/rainbond/issues) --- ## Contributing If you want to contribute, start here: - [Contribution Guide](https://www.rainbond.com/docs/contribution?channel=github) - [Rainbond Docs](https://github.com/goodrain/rainbond-docs) - [Open an Issue](https://github.com/goodrain/rainbond/issues) We welcome: - Code contributions - Documentation improvements - User stories and technical writeups - Application template and plugin sharing --- ## Related projects - [rainbond-console](https://github.com/goodrain/rainbond-console) - Console backend - [rainbond-ui](https://github.com/goodrain/rainbond-ui) - Console frontend - [rainbond-operator](https://github.com/goodrain/rainbond-operator) - Installation and operations - [builder](https://github.com/goodrain/builder) - Source build toolset --- ## License This repository is licensed under the [Rainbond Open Source License](./LICENSE), based on Apache 2.0 with additional conditions. --- ### Overview (`LICENSES/README.md`) # Third-Party Licenses This directory contains license files for third-party components used in this project. ## Directory Structure - `apache/` - Apache Software Foundation licensed components - `apecloud/` - ApeCloud licensed components - `minio/` - MinIO licensed components - `rancher/` - Rancher licensed components ## Purpose These licenses are maintained to ensure compliance with the licensing requirements of all third-party software integrated into this project. --- ### Overview (`plugins/README.md`) ## Rainbond Plugins This directory contains various plugin components of the Rainbond platform, which are used to expand and enhance the functionality of Rainbond. ## License This project follows the same license as the Rainbond main project. --- ### Overview (`third_party/coreos-etcd/README.md`) # etcd [](https://goreportcard.com/report/github.com/coreos/etcd) [](https://codecov.io/gh/coreos/etcd) [](https://travis-ci.org/coreos/etcd) [](https://semaphoreci.com/coreos/etcd) [](https://godoc.org/github.com/coreos/etcd) [](https://github.com/coreos/etcd/releases) [](https://github.com/coreos/etcd/blob/master/LICENSE) **Note**: The `master` branch may be in an *unstable or even broken state* during development. Please use [releases][github-release] instead of the `master` branch in order to get stable binaries. *the etcd v2 [documentation](Documentation/v2/README.md) has moved* etcd is a distributed reliable key-value store for the most critical data of a distributed system, with a focus on being: * *Simple*: well-defined, user-facing API (gRPC) * *Secure*: automatic TLS with optional client cert authentication * *Fast*: benchmarked 10,000 writes/sec * *Reliable*: properly distributed using Raft etcd is written in Go and uses the [Raft][raft] consensus algorithm to manage a highly-available replicated log. etcd is used [in production by many companies](./Documentation/production-users.md), and the development team stands behind it in critical deployment scenarios, where etcd is frequently teamed with applications such as [Kubernetes][k8s], [fleet][fleet], [locksmith][locksmith], [vulcand][vulcand], [Doorman][doorman], and many others. Reliability is further ensured by rigorous [testing][etcd-tests]. See [etcdctl][etcdctl] for a simple command line client. [raft]: https://raft.github.io/ [k8s]: http://kubernetes.io/ [doorman]: https://github.com/youtube/doorman [fleet]: https://github.com/coreos/fleet [locksmith]: https://github.com/coreos/locksmith [vulcand]: https://github.com/vulcand/vulcand [etcdctl]: https://github.com/coreos/etcd/tree/master/etcdctl [etcd-tests]: http://dash.etcd.io ## Community meetings etcd contributors and maintainers have bi-weekly meetings at 11:00 AM (USA Pacific) on Tuesdays. There is an [iCalendar][rfc5545] format for the meetings [here](meeting.ics). Anyone is welcome to join via [Zoom][zoom] or audio-only: +1 669 900 6833. An initial agenda will be posted to the [shared Google docs][shared-meeting-notes] a day before each meeting, and everyone is welcome to suggest additional topics or other agendas. [rfc5545]: https://tools.ietf.org/html/rfc5545 [zoom]: https://coreos.zoom.us/j/854793406 [shared-meeting-notes]: https://docs.google.com/document/d/1DbVXOHvd9scFsSmL2oNg4YGOHJdXqtx583DmeVWrB_M/edit# ## Getting started ### Getting etcd The easiest way to get etcd is to use one of the pre-built release binaries which are available for OSX, Linux, Windows, [rkt][rkt], and Docker. Instructions for using these binaries are on the [GitHub releases page][github-release]. For those wanting to try the very latest version, [build the latest version of etcd][dl-build] from the `master` branch. This first needs [*Go*](https://golang.org/) installed (version 1.9+ is required). All development occurs on `master`, including new features and bug fixes. Bug fixes are first targeted at `master` and subsequently ported to release branches, as described in the [branch management][branch-management] guide. [rkt]: https://github.com/rkt/rkt/releases/ [github-release]: https://github.com/coreos/etcd/releases/ [branch-management]: ./Documentation/branch_management.md [dl-build]: ./Documentation/dl_build.md#build-the-latest-version ### Running etcd First start a single-member cluster of etcd. If etcd is installed using the [pre-built release binaries][github-release], run it from the installation location as below: ```sh /tmp/etcd-download-test/etcd ``` The etcd command can be simply run as such if it is moved to the system path as below: ```sh mv /tmp/etcd-download-test/etcd /usr/locale/bin/ etcd ``` If etcd is [build from the master branch][dl-build], run it as below: ```sh ./bin/etcd ``` This will bring up etcd listening on port 2379 for client communication and on port 2380 for server-to-server communication. Next, let's set a single key, and then retrieve it: ``` ETCDCTL_API=3 etcdctl put mykey "this is awesome" ETCDCTL_API=3 etcdctl get mykey ``` That's it! etcd is now running and serving client requests. For more - [Animated quick demo][demo-gif] - [Interactive etcd playground][etcd-play] [demo-gif]: ./Documentation/demo.md [etcd-play]: http://play.etcd.io/ ### etcd TCP ports The [official etcd ports][iana-ports] are 2379 for client requests, and 2380 for peer communication. [iana-ports]: http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt ### Running a local etcd cluster First install [goreman](https://github.com/mattn/goreman), which manages Procfile-based applications. Our [Procfile script](./Procfile) will set up a local example cluster. Start it with: ```sh goreman start ``` This will bring up 3 etcd members `infra1`, `infra2` and `infra3` and etcd `grpc-proxy`, which runs locally and composes a cluster. Every cluster member and proxy accepts key value reads and key value writes. ### Running etcd on Kubernetes To run an etcd cluster on Kubernetes, try [etcd operator](https://github.com/coreos/etcd-operator). ### Next steps Now it's time to dig into the full etcd API and other guides. - Read the full [documentation][fulldoc]. - Explore the full gRPC [API][api]. - Set up a [multi-machine cluster][clustering]. - Learn the [config format, env variables and flags][configuration]. - Find [language bindings and tools][integrations]. - Use TLS to [secure an etcd cluster][security]. - [Tune etcd][tuning]. [fulldoc]: ./Documentation/docs.md [api]: ./Documentation/dev-guide/api_reference_v3.md [clustering]: ./Documentation/op-guide/clustering.md [configuration]: ./Documentation/op-guide/configuration.md [integrations]: ./Documentation/integrations.md [security]: ./Documentation/op-guide/security.md [tuning]: ./Documentation/tuning.md ## Contact - Mailing list: [etcd-dev](https://groups.google.com/forum/?hl=en#!forum/etcd-dev) - IRC: #[etcd](irc://irc.freenode.org:6667/#etcd) on freenode.org - Planning/Roadmap: [milestones](https://github.com/coreos/etcd/milestones), [roadmap](./ROADMAP.md) - Bugs: [issues](https://github.com/coreos/etcd/issues) ## Contributing See [CONTRIBUTING](CONTRIBUTING.md) for details on submitting patches and the contribution workflow. ## Reporting bugs See [reporting bugs](Documentation/reporting_bugs.md) for details about reporting any issues. ### License etcd is under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details. --- ### Overview (`third_party/process-exporter/README.md`) # process-exporter Prometheus exporter that mines /proc to report on selected processes. [release]: https://github.com/ncabatoff/process-exporter/releases/latest [][release] [](https://github.com/goreleaser) [](https://circleci.com/gh/ncabatoff/process-exporter) Some apps are impractical to instrument directly, either because you don't control the code or they're written in a language that isn't easy to instrument with Prometheus. We must instead resort to mining /proc. ## Installation Either grab a package for your OS from the [Releases][release] page, or install via [docker](https://hub.docker.com/r/ncabatoff/process-exporter/). ## Running Usage: ``` process-exporter [options] -config.path filename.yml ``` or via docker: ``` docker run -d --rm -p 9256:9256 --privileged -v /proc:/host/proc -v `pwd`:/config ncabatoff/process-exporter --procfs /host/proc -config.path /config/filename.yml ``` Important options (run process-exporter --help for full list): -children (default:true) makes it so that any process that otherwise isn't part of its own group becomes part of the first group found (if any) when walking the process tree upwards. In other words, resource usage of subprocesses is added to their parent's usage unless the subprocess identifies as a different group name. -threads (default:true) means that metrics will be broken down by thread name as well as group name. -recheck (default:false) means that on each scrape the process names are re-evaluated. This is disabled by default as an optimization, but since processes can choose to change their names, this may result in a process falling into the wrong group if we happen to see it for the first time before it's assumed its proper name. -procnames is intended as a quick alternative to using a config file. Details in the following section. To disable any of these options, use the `-option=false`. ## Configuration and group naming To select and group the processes to monitor, either provide command-line arguments or use a YAML configuration file. The recommended option is to use a config file via -config.path, but for convenience and backwards compatibility the -procnames/-namemapping options exist as an alternative. ### Using a config file The general format of the -config.path YAML file is a top-level `process_names` section, containing a list of name matchers: ``` process_names: - matcher1 - matcher2 ... - matcherN ``` The default config shipped with the deb/rpm packages is: ``` process_names: - name: "{{.Comm}}" cmdline: - '.+' ``` A process may only belong to one group: even if multiple items would match, the first one listed in the file wins. (Side note: to avoid confusion with the cmdline YAML element, we'll refer to the command-line arguments of a process `/proc//cmdline` as the array `argv[]`.) #### Using a config file: group name Each item in `process_names` gives a recipe for identifying and naming processes. The optional `name` tag defines a template to use to name matching processes; if not specified, `name` defaults to `{{.ExeBase}}`. Template variables available: - `{{.Comm}}` contains the basename of the original executable, i.e. 2nd field in `/proc//stat` - `{{.ExeBase}}` contains the basename of the executable - `{{.ExeFull}}` contains the fully qualified path of the executable - `{{.Username}}` contains the username of the effective user - `{{.Matches}}` map contains all the matches resulting from applying cmdline regexps - `{{.PID}}` contains the PID of the process. Note that using PID means the group will only contain a single process. - `{{.StartTime}}` contains the start time of the process. This can be useful in conjunction with PID because PIDs get reused over time. Using `PID` or `StartTime` is discouraged: this is almost never what you want, and is likely to result in high cardinality metrics which Prometheus will have trouble with. #### Using a config file: process selectors Each item in `process_names` must contain one or more selectors (`comm`, `exe` or `cmdline`); if more than one selector is present, they must all match. Each selector is a list of strings to match against a process's `comm`, `argv[0]`, or in the case of `cmdline`, a regexp to apply to the command line. The cmdline regexp uses the [Go syntax](https://golang.org/pkg/regexp). For `comm` and `exe`, the list of strings is an OR, meaning any process matching any of the strings will be added to the item's group. For `cmdline`, the list of regexes is an AND, meaning they all must match. Any capturing groups in a regexp must use the `?P` option to assign a name to the capture, which is used to populate `.Matches`. Performance tip: give an exe or comm clause in addition to any cmdline clause, so you avoid executing the regexp when the executable name doesn't match. ``` process_names: # comm is the second field of /proc//stat minus parens. # It is the base executable name, truncated at 15 chars. # It cannot be modified by the program, unlike exe. - comm: - bash # exe is argv[0]. If no slashes, only basename of argv[0] need match. # If exe contains slashes, argv[0] must match exactly. - exe: - postgres - /usr/local/bin/prometheus # cmdline is a list of regexps applied to argv. # Each must match, and any captures are added to the .Matches map. - name: "{{.ExeFull}}:{{.Matches.Cfgfile}}" exe: - /usr/local/bin/process-exporter cmdline: - -config.path\s+(?P\S+) ``` Here's the config I use on my home machine: ``` process_names: - comm: - chromium-browse - bash - prometheus - gvim - exe: - /sbin/upstart cmdline: - --user name: upstart:-user ``` ### Using -procnames/-namemapping instead of config.path Every name in the procnames list becomes a process group. The default name of a process is the value found in the second field of /proc//stat ("comm"), which is truncated at 15 chars. Usually this is the same as the name of the executable. If -namemapping isn't provided, every process with a comm value present in -procnames is assigned to a group based on that name, and any other processes are ignored. The -namemapping option is a comma-separated list of alternating name,regexp values. It allows assigning a name to a process based on a combination of the process name and command line. For example, using -namemapping "python2,([^/]+)\.py,java,-jar\s+([^/]+).jar" will make it so that each different python2 and java -jar invocation will be tracked with distinct metrics. Processes whose remapped name is absent from the procnames list will be ignored. On a Ubuntu Xenian machine being used as a workstation, here's a good way of tracking resource usage for a few different key user apps: process-exporter -namemapping "upstart,(--user)" \ -procnames chromium-browse,bash,gvim,prometheus,process-exporter,upstart:-user Since upstart --user is the parent process of the X11 session, this will make all apps started by the user fall into the group named "upstart:-user", unless they're one of the others named explicitly with -procnames, like gvim. ## Group Metrics There's no meaningful way to name a process that will only ever name a single process, so process-exporter assumes that every metric will be attached to a group of processes - not a [process group](https://en.wikipedia.org/wiki/Process_group) in the technical sense, just one or more processes that meet a configuration's specification of what should be monitored and how to name it. All these metrics start with `namedprocess_namegroup_` and have at minimum the label `groupname`. ### num_procs gauge Number of processes in this group. ### cpu_seconds_total counter CPU usage based on /proc/[pid]/stat fields utime(14) and stime(15) i.e. user and system time. This is similar to the node\_exporter's `node_cpu_seconds_total`. ### read_bytes_total counter Bytes read based on /proc/[pid]/io field read_bytes. The man page says > Attempt to count the number of bytes which this process really did cause to be fetched from the storage layer. This is accurate for block-backed filesystems. but I would take it with a grain of salt. ### write_bytes_total counter Bytes written based on /proc/[pid]/io field write_bytes. As with read_bytes, somewhat dubious. May be useful for isolating which processes are doing the most I/O, but probably not measuring just how much I/O is happening. ### major_page_faults_total counter Number of major page faults based on /proc/[pid]/stat field majflt(12). ### minor_page_faults_total counter Number of minor page faults based on /proc/[pid]/stat field minflt(10). ### context_switches_total counter Number of context switches based on /proc/[pid]/status fields voluntary_ctxt_switches and nonvoluntary_ctxt_switches. The extra label `ctxswitchtype` can have two values: `voluntary` and `nonvoluntary`. ### memory_bytes gauge Number of bytes of memory used. The extra label `memtype` can have three values: *resident*: Field rss(24) from /proc/[pid]/stat, whose doc says: > This is just the pages which count toward text, data, or stack space. This does not include pages which have not been demand-loaded in, or which are swapped out. *virtual*: Field vsize(23) from /proc/[pid]/stat, virtual memory size. *swapped*: Field VmSwap from /proc/[pid]/status, translated from KB to bytes. If gathering smaps file is enabled, two additional values for `memtype` are added: *proportionalResident*: Sum of "Pss" fields from /proc/[pid]/smaps, whose doc says: > The "proportional set size" (PSS) of a process is the count of pages it has > in memory, where each page is divided by the number of processes sharing it. *proportionalSwapped*: Sum of "SwapPss" fields from /proc/[pid]/smaps ### open_filedesc gauge Number of file descriptors, based on counting how many entries are in the directory /proc/[pid]/fd. ### worst_fd_ratio gauge Worst ratio of open filedescs to filedesc limit, amongst all the procs in the group. The limit is the fd soft limit based on /proc/[pid]/limits. Normally Prometheus metrics ought to be as "basic" as possible (i.e. the raw values rather than a derived ratio), but we use a ratio here because nothing else makes sense. Suppose there are 10 procs in a given group, each with a soft limit of 4096, and one of them has 4000 open fds and the others all have 40, their total fdcount is 4360 and total soft limit is 40960, so the ratio is 1:10, but in fact one of the procs is about to run out of fds. With worst_fd_ratio we're able to know this: in the above example it would be 0.97, rather than the 0.10 you'd see if you computed sum(open_filedesc) / sum(limit_filedesc). ### oldest_start_time_seconds gauge Epoch time (seconds since 1970/1/1) at which the oldest process in the group started. This is derived from field starttime(22) from /proc/[pid]/stat, added to boot time to make it relative to epoch. ### num_threads gauge Sum of number of threads of all process in the group. Based on field num_threads(20) from /proc/[pid]/stat. ### states gauge Number of threads in the group in each of various states, based on the field state(3) from /proc/[pid]/stat. The extra label `state` can have these values: `Running`, `Sleeping`, `Waiting`, `Zombie`, `Other`. ## Group Thread Metrics Since publishing thread metrics adds a lot of overhead, use the `-threads` command-line argument to disable them, if necessary. All these metrics start with `namedprocess_namegroup_` and have at minimum the labels `groupname` and `threadname`. `threadname` is field comm(2) from /proc/[pid]/stat. Just as groupname breaks the set of processes down into groups, threadname breaks a given process group down into subgroups. ### thread_count gauge Number of threads in this thread subgroup. ### thread_cpu_seconds_total counter Same as cpu_user_seconds_total and cpu_system_seconds_total, but broken down per-thread subgroup. Unlike cpu_user_seconds_total/cpu_system_seconds_total, the label `cpumode` is used to distinguish between `user` and `system` time. ### thread_io_bytes_total counter Same as read_bytes_total and write_bytes_total, but broken down per-thread subgroup. Unlike read_bytes_total/write_bytes_total, the label `iomode` is used to distinguish between `read` and `write` bytes. ### thread_major_page_faults_total counter Same as major_page_faults_total, but broken down per-thread subgroup. ### thread_minor_page_faults_total counter Same as minor_page_faults_total, but broken down per-thread subgroup. ### thread_context_switches_total counter Same as context_switches_total, but broken down per-thread subgroup. ## Instrumentation cost process-exporter will consume CPU in proportion to the number of processes in the system and the rate at which new ones are created. The most expensive parts - applying regexps and executing templates - are only applied once per process seen, unless the command-line option -recheck is provided. If you have mostly long-running processes process-exporter overhead should be minimal: each time a scrape occurs, it will parse of /proc/$pid/stat and /proc/$pid/cmdline for every process being monitored and add a few numbers. ## Dashboards An example Grafana dashboard to view the metrics is available at https://grafana.net/dashboards/249 ## Building Requires Go 1.12 installed. ``` make ``` --- ### Overview (`util/http/README.md`) ### 用法实例 ``` type User struct { Username string `validate:"username|required|between:3,8"` Email string `validate:"email|email"` Web string `validate:"web|url"` Age int `validate:"age|numeric_between:18,55"` } func handler(w http.ResponseWriter, r *http.Request) { var user User messages := validator.MapData{ "username": []string{"required:You must provide username", "between:username must be between 3 to 8 chars"}, "web": []string{"url:You must provide a valid url"}, } if !httputil.ValidatorRequestStructAndErrorResponse(r, w, &user, messages) { return } httputil.ReturnSuccess(r, w, user) } func handlerMap(w http.ResponseWriter, r *http.Request) { rule := validator.MapData{ "username": []string{"required", "between:3,5"}, "web": []string{"url"}, } messages := validator.MapData{ "username": []string{"required:You must provide username", "between:username must be between 3 to 8 chars"}, "web": []string{"url:You must provide a valid url"}, } data, ok := httputil.ValidatorRequestMapAndErrorResponse(r, w, rule, messages) if !ok { return } httputil.ReturnSuccess(r, w, data) } ``` ### Validation Rules * `alpha` The field under validation must be entirely alphabetic characters. * `alpha_dash` The field under validation may have alpha-numeric characters, as well as dashes and underscores. * `alpha_num` The field under validation must be entirely alpha-numeric characters. * `numeric` The field under validation must be entirely numeric characters. * `numeric_between:int,int` The field under validation must be a numeric value between the range. e.g: `numeric_between:18,65` may contains numeric value like `35`, `55` * `bool` The field under validation must be able to be cast as a boolean. Accepted input are `true, false, 1, 0, "1" and "0"`. * `credit_card` The field under validation must have a valid credit card number. Accepted cards are `Visa, MasterCard, American Express, Diners Club, Discover and JCB card` * `coordinate` The field under validation must have a value of valid coordinate. * `css_color` The field under validation must have a value of valid CSS color. Accepted colors are `hex, rgb, rgba, hsl, hsla` like `#909, #00aaff, rgb(255,122,122)` * `date` The field under validation must have a valid date of format yyyy-mm-dd or yyyy/mm/dd. * `date:dd-mm-yyyy` The field under validation must have a valid date of format dd-mm-yyyy. * `digits:int` The field under validation must be numeric and must have an exact length of value. * `digits_between:int,int` The field under validation must be numeric and must have length between the range. e.g: `digits_between:3,5` may contains digits like `2323`, `12435` * `email` The field under validation must have a valid email. * `float` The field under validation must have a valid float number. * `in:foo,bar` The field under validation must have one of the values. e.g: `in:admin,manager,user` must contain the values (admin or manager or user) * `min:int` The field under validation must have a min length of characters. e.g: `min:3` may contains characters minimum length of 3 like `"john", "jane", "jane321"` but not `"mr", "xy"` * `max:int` The field under validation must have a max length of characters. e.g: `max:6` may contains characters maximum length of 6 like `"john doe", "jane doe"` but not `"john", "jane"` * `not_in:foo,bar` The field under validation must have one value except foo,bar. e.g: `not_in:admin,manager,user` must not contain the values (admin or manager or user) * `len:int` The field under validation must have an exact length of characters. e.g: `len:4` may contains characters exact length of 4 like `Food, Mood, Good` * `ip` The field under validation must be a valid IP address. * `ip_v4` The field under validation must be a valid IP V4 address. * `ip_v6` The field under validation must be a valid IP V6 address. * `json` The field under validation must be a valid JSON string. * `lat` The field under validation must be a valid latitude. * `lon` The field under validation must be a valid longitude. * `regex:regurlar expression` The field under validation validate against the regex. e.g: `regex:^[a-zA-Z]+$` validate the letters. * `required` The field under validation must be present in the input data and not empty. A field is considered "empty" if one of the following conditions are true: 1) The value is null. 2)The value is an empty string. * `url` The field under validation must be a valid URL. * `uuid` The field under validation must be a valid UUID. * `uuid_v3` The field under validation must be a valid UUID V3. * `uuid_v4` The field under validation must be a valid UUID V4. * `uuid_v5` The field under validation must be a valid UUID V5. --- ### Overview (`util/termtables/README.md`) # Termtables [](https://travis-ci.org/apcera/termtables) A [Go](http://golang.org) port of the Ruby library [terminal-tables](https://github.com/visionmedia/terminal-table) for fast and simple ASCII table generation. ## Installation ```bash go get github.com/apcera/termtables ``` ## Go Style Documentation [http://godoc.org/github.com/apcera/termtables](http://godoc.org/github.com/apcera/termtables) ## APC Command Line usage `--markdown` — output a markdown table, e.g. `apc app list --markdown` `--html` — output an html table, e.g. `apc app list --html` `--ascii` — output an ascii table, e.g. `apc app list --ascii` ## Basic Usage ```go package main import ( "fmt" "github.com/apcera/termtables" ) func main() { table := termtables.CreateTable() table.AddHeaders("Name", "Age") table.AddRow("John", "30") table.AddRow("Sam", 18) table.AddRow("Julie", 20.14) fmt.Println(table.Render()) } ``` Result: ``` +-------+-------+ | Name | Age | +-------+-------+ | John | 30 | | Sam | 18 | | Julie | 20.14 | +-------+-------+ ``` ## Advanced Usage The package function-call `EnableUTF8()` will cause any tables created after that point to use Unicode box-drawing characters for the table lines. Calling `EnableUTF8PerLocale()` uses the C library's locale functionality to determine if the current locale environment variables say that the current character map is UTF-8. If, and only if, so, then `EnableUTF8()` will be called. Calling `SetModeHTML(true)` will cause any tables created after that point to be emitted in HTML, while `SetModeMarkdown(true)` will trigger Markdown. Neither should result in changes to later API to get the different results; the primary intended use-case is extracting the same table, but for documentation. The table method `.AddSeparator()` inserts a rule line in the output. This only applies in normal terminal output mode. The table method `.AddTitle()` adds a title to the table; in terminal output, this is an initial row; in HTML, it's a caption. In Markdown, it's a line of text before the table, prefixed by `Table: `. The table method `.SetAlign()` takes an alignment and a column number (indexing starts at 1) and changes all _current_ cells in that column to have the given alignment. It does not change the alignment of cells added to the table after this call. Alignment is only stored on a per-cell basis. ## Known Issues Normal output: * `.SetAlign()` does not affect headers. Markdown output mode: * When emitting Markdown, the column markers are not re-flowed if a vertical bar is an element of a cell, causing an escape to take place; since Markdown is often converted to HTML, this only affects text viewing. * A title in Markdown is not escaped against all possible forms of Markdown markup (to avoid adding a dependency upon a Markdown library, as supported syntax can vary). * Markdown requires headers, so a dummy header will be inserted if needed. * Table alignment is not reflected in Markdown output. --- ### Overview (`hack/k8s/codegen/README.md`) ## How to use codegen ```sh ./hack/k8s/codegen/update-generated.sh ``` It should print: ``` Generating deepcopy funcs Generating clientset for etcd:v1beta2 at github.com/coreos/etcd-operator/pkg/generated/clientset Generating listers for etcd:v1beta2 at github.com/coreos/etcd-operator/pkg/generated/listers Generating informers for etcd:v1beta2 at github.com/coreos/etcd-operator/pkg/generated/informers ``` --- ### Overview (`third_party/coreos-etcd/client/README.md`) # etcd/client etcd/client is the Go client library for etcd. [](https://godoc.org/github.com/coreos/etcd/client) etcd uses `cmd/vendor` directory to store external dependencies, which are to be compiled into etcd release binaries. `client` can be imported without vendoring. For full compatibility, it is recommended to vendor builds using etcd's vendored packages, using tools like godep, as in [vendor directories](https://golang.org/cmd/go/#hdr-Vendor_Directories). For more detail, please read [Go vendor design](https://golang.org/s/go15vendor). ## Install ```bash go get github.com/coreos/etcd/client ``` ## Usage ```go package main import ( "log" "time" "context" "github.com/coreos/etcd/client" ) func main() { cfg := client.Config{ Endpoints: []string{"http://127.0.0.1:2379"}, Transport: client.DefaultTransport, // set timeout per request to fail fast when the target endpoint is unavailable HeaderTimeoutPerRequest: time.Second, } c, err := client.New(cfg) if err != nil { log.Fatal(err) } kapi := client.NewKeysAPI(c) // set "/foo" key with "bar" value log.Print("Setting '/foo' key with 'bar' value") resp, err := kapi.Set(context.Background(), "/foo", "bar", nil) if err != nil { log.Fatal(err) } else { // print common key info log.Printf("Set is done. Metadata is %q\n", resp) } // get "/foo" key's value log.Print("Getting '/foo' key value") resp, err = kapi.Get(context.Background(), "/foo", nil) if err != nil { log.Fatal(err) } else { // print common key info log.Printf("Get is done. Metadata is %q\n", resp) // print value log.Printf("%q key has %q value\n", resp.Node.Key, resp.Node.Value) } } ``` ## Error Handling etcd client might return three types of errors. - context error Each API call has its first parameter as `context`. A context can be canceled or have an attached deadline. If the context is canceled or reaches its deadline, the responding context error will be returned no matter what internal errors the API call has already encountered. - cluster error Each API call tries to send request to the cluster endpoints one by one until it successfully gets a response. If a requests to an endpoint fails, due to exceeding per request timeout or connection issues, the error will be added into a list of errors. If all possible endpoints fail, a cluster error that includes all encountered errors will be returned. - response error If the response gets from the cluster is invalid, a plain string error will be returned. For example, it might be a invalid JSON error. Here is the example code to handle client errors: ```go cfg := client.Config{Endpoints: []string{"http://etcd1:2379","http://etcd2:2379","http://etcd3:2379"}} c, err := client.New(cfg) if err != nil { log.Fatal(err) } kapi := client.NewKeysAPI(c) resp, err := kapi.Set(ctx, "test", "bar", nil) if err != nil { if err == context.Canceled { // ctx is canceled by another routine } else if err == context.DeadlineExceeded { // ctx is attached with a deadline and it exceeded } else if cerr, ok := err.(*client.ClusterError); ok { // process (cerr.Errors) } else { // bad cluster endpoints, which are not etcd servers } } ``` ## Caveat 1. etcd/client prefers to use the same endpoint as long as the endpoint continues to work well. This saves socket resources, and improves efficiency for both client and server side. This preference doesn't remove consistency from the data consumed by the client because data replicated to each etcd member has already passed through the consensus process. 2. etcd/client does round-robin rotation on other available endpoints if the preferred endpoint isn't functioning properly. For example, if the member that etcd/client connects to is hard killed, etcd/client will fail on the first attempt with the killed member, and succeed on the second attempt with another member. If it fails to talk to all available endpoints, it will return all errors happened. 3. Default etcd/client cannot handle the case that the remote server is SIGSTOPed now. TCP keepalive mechanism doesn't help in this scenario because operating system may still send TCP keep-alive packets. Over time we'd like to improve this functionality, but solving this issue isn't high priority because a real-life case in which a server is stopped, but the connection is kept alive, hasn't been brought to our attention. 4. etcd/client cannot detect whether a member is healthy with watches and non-quorum read requests. If the member is isolated from the cluster, etcd/client may retrieve outdated data. Instead, users can either issue quorum read requests or monitor the /health endpoint for member health information. --- ### Overview (`third_party/coreos-etcd/clientv3/README.md`) # etcd/clientv3 [](https://godoc.org/github.com/coreos/etcd/clientv3) `etcd/clientv3` is the official Go etcd client for v3. ## Install ```bash go get github.com/coreos/etcd/clientv3 ``` ## Get started Create client using `clientv3.New`: ```go cli, err := clientv3.New(clientv3.Config{ Endpoints: []string{"localhost:2379", "localhost:22379", "localhost:32379"}, DialTimeout: 5 * time.Second, }) if err != nil { // handle error! } defer cli.Close() ``` etcd v3 uses [`gRPC`](http://www.grpc.io) for remote procedure calls. And `clientv3` uses [`grpc-go`](https://github.com/grpc/grpc-go) to connect to etcd. Make sure to close the client after using it. If the client is not closed, the connection will have leaky goroutines. To specify client request timeout, pass `context.WithTimeout` to APIs: ```go ctx, cancel := context.WithTimeout(context.Background(), timeout) resp, err := cli.Put(ctx, "sample_key", "sample_value") cancel() if err != nil { // handle error! } // use the response ``` etcd uses `cmd/vendor` directory to store external dependencies, which are to be compiled into etcd release binaries. `client` can be imported without vendoring. For full compatibility, it is recommended to vendor builds using etcd's vendored packages, using tools like godep, as in [vendor directories](https://golang.org/cmd/go/#hdr-Vendor_Directories). For more detail, please read [Go vendor design](https://golang.org/s/go15vendor). ## Error Handling etcd client returns 2 types of errors: 1. context error: canceled or deadline exceeded. 2. gRPC error: see [api/v3rpc/rpctypes](https://godoc.org/github.com/coreos/etcd/etcdserver/api/v3rpc/rpctypes). Here is the example code to handle client errors: ```go resp, err := cli.Put(ctx, "", "") if err != nil { switch err { case context.Canceled: log.Fatalf("ctx is canceled by another routine: %v", err) case context.DeadlineExceeded: log.Fatalf("ctx is attached with a deadline is exceeded: %v", err) case rpctypes.ErrEmptyKey: log.Fatalf("client-side error: %v", err) default: log.Fatalf("bad cluster endpoints, which are not etcd servers: %v", err) } } ``` ## Metrics The etcd client optionally exposes RPC metrics through [go-grpc-prometheus](https://github.com/grpc-ecosystem/go-grpc-prometheus). See the [examples](https://github.com/coreos/etcd/blob/master/clientv3/example_metrics_test.go). ## Namespacing The [namespace](https://godoc.org/github.com/coreos/etcd/clientv3/namespace) package provides `clientv3` interface wrappers to transparently isolate client requests to a user-defined prefix. ## Examples More code examples can be found at [GoDoc](https://godoc.org/github.com/coreos/etcd/clientv3). --- ### Overview (`third_party/coreos-etcd/cmd/README.md`) ## cmd This directory is meant to enforce vendoring for etcd binaries without polluting the etcd client libraries with vendored dependencies. --- ### Overview (`third_party/coreos-etcd/contrib/README.md`) ## Contrib Scripts and files which may be useful but aren't part of the core etcd project. * [systemd](systemd) - an example unit file for deploying etcd on systemd-based distributions * [raftexample](raftexample) - an example distributed key-value store using raft * [systemd/etcd2-backup-coreos](systemd/etcd2-backup-coreos) - remote backup and restore procedures for etcd2 clusters on CoreOS Linux * [systemd/etcd3-multinode](systemd/etcd3-multinode) - multi-node cluster setup with systemd --- ### Overview (`third_party/coreos-etcd/etcdctl/README.md`) etcdctl ======== `etcdctl` is a command line client for [etcd][etcd]. Make sure to set environment variable `ETCDCTL_API=3`. For etcdctl v2, please check [READMEv2][READMEv2]. Global flags (e.g., `dial-timeout`, `--cacert`, `--cert`, `--key`) can be set with environment variables: ``` ETCDCTL_DIAL_TIMEOUT=3s ETCDCTL_CACERT=/tmp/ca.pem ETCDCTL_CERT=/tmp/cert.pem ETCDCTL_KEY=/tmp/key.pem ``` Prefix flag strings with `ETCDCTL_`, convert all letters to upper-case, and replace dash(`-`) with underscore(`_`). ## Key-value commands ### PUT [options] \ \ PUT assigns the specified value with the specified key. If key already holds a value, it is overwritten. RPC: Put #### Options - lease -- lease ID (in hexadecimal) to attach to the key. - prev-kv -- return the previous key-value pair before modification. - ignore-value -- updates the key using its current value. - ignore-lease -- updates the key using its current lease. #### Output `OK` #### Examples ```bash ./etcdctl put foo bar --lease=1234abcd # OK ./etcdctl get foo # foo # bar ./etcdctl put foo --ignore-value # to detache lease # OK ``` ```bash ./etcdctl put foo bar --lease=1234abcd # OK ./etcdctl put foo bar1 --ignore-lease # to use existing lease 1234abcd # OK ./etcdctl get foo # foo # bar1 ``` ```bash ./etcdctl put foo bar1 --prev-kv # OK # foo # bar ./etcdctl get foo # foo # bar1 ``` #### Remarks If \ isn't given as command line argument, this command tries to read the value from standard input. When \ begins with '-', \ is interpreted as a flag. Insert '--' for workaround: ```bash ./etcdctl put -- ./etcdctl put -- ``` ### GET [options] \ [range_end] GET gets the key or a range of keys [key, range_end) if `range-end` is given. RPC: Range #### Options - hex -- print out key and value as hex encode string - limit -- maximum number of results - prefix -- get keys by matching prefix - order -- order of results; ASCEND or DESCEND - sort-by -- sort target; CREATE, KEY, MODIFY, VALUE, or VERSION - rev -- specify the kv revision - print-value-only -- print only value when used with write-out=simple - consistency -- Linearizable(l) or Serializable(s) - from-key -- Get keys that are greater than or equal to the given key using byte compare - keys-only -- Get only the keys #### Output \\n\\n\\n\... #### Examples First, populate etcd with some keys: ```bash ./etcdctl put foo bar # OK ./etcdctl put foo1 bar1 # OK ./etcdctl put foo2 bar2 # OK ./etcdctl put foo3 bar3 # OK ``` Get the key named `foo`: ```bash ./etcdctl get foo # foo # bar ``` Get all keys: ```bash ./etcdctl get --from-key '' # foo # bar # foo1 # bar1 # foo2 # foo2 # foo3 # bar3 ``` Get all keys with names greater than or equal to `foo1`: ```bash ./etcdctl get --from-key foo1 # foo1 # bar1 # foo2 # bar2 # foo3 # bar3 ``` Get keys with names greater than or equal to `foo1` and less than `foo3`: ```bash ./etcdctl get foo1 foo3 # foo1 # bar1 # foo2 # bar2 ``` #### Remarks If any key or value contains non-printable characters or control characters, simple formatted output can be ambiguous due to new lines. To resolve this issue, set `--hex` to hex encode all strings. ### DEL [options] \ [range_end] Removes the specified key or range of keys [key, range_end) if `range-end` is given. RPC: DeleteRange #### Options - prefix -- delete keys by matching prefix - prev-kv -- return deleted key-value pairs - from-key -- delete keys that are greater than or equal to the given key using byte compare #### Output Prints the number of keys that were removed in decimal if DEL succeeded. #### Examples ```bash ./etcdctl put foo bar # OK ./etcdctl del foo # 1 ./etcdctl get foo ``` ```bash ./etcdctl put key val # OK ./etcdctl del --prev-kv key # 1 # key # val ./etcdctl get key ``` ```bash ./etcdctl put a 123 # OK ./etcdctl put b 456 # OK ./etcdctl put z 789 # OK ./etcdctl del --from-key a # 3 ./etcdctl get --from-key a ``` ```bash ./etcdctl put zoo val # OK ./etcdctl put zoo1 val1 # OK ./etcdctl put zoo2 val2 # OK ./etcdctl del --prefix zoo # 3 ./etcdctl get zoo2 ``` ### TXN [options] TXN reads multiple etcd requests from standard input and applies them as a single atomic transaction. A transaction consists of list of conditions, a list of requests to apply if all the conditions are true, and a list of requests to apply if any condition is false. RPC: Txn #### Options - hex -- print out keys and values as hex encoded strings. - interactive -- input transaction with interactive prompting. #### Input Format ```ebnf ::= * "\n" "\n" "\n" ::= (||||) "\n" ::= "<" | "=" | ">" := ("c"|"create")"("")" ::= ("m"|"mod")"("")" ::= ("val"|"value")"("")" ::= ("ver"|"version")"("")" ::= "lease("")" ::= * ::= * ::= ((see put, get, del etcdctl command syntax)) "\n" ::= (%q formatted string) ::= (%q formatted string) ::= "\""[0-9]+"\"" ::= "\""[0-9]+"\"" ::= "\""[0-9]+\"" ``` #### Output `SUCCESS` if etcd processed the transaction success list, `FAILURE` if etcd processed the transaction failure list. Prints the output for each command in the executed request list, each separated by a blank line. #### Examples txn in interactive mode: ```bash ./etcdctl txn -i # compares: mod("key1") > "0" # success requests (get, put, delete): put key1 "overwrote-key1" # failure requests (get, put, delete): put key1 "created-key1" put key2 "some extra key" # FAILURE # OK # OK ``` txn in non-interactive mode: ```bash ./etcdctl txn <<<'mod("key1") > "0" put key1 "overwrote-key1" put key1 "created-key1" put key2 "some extra key" ' # FAILURE # OK # OK ``` ### COMPACTION [options] \ COMPACTION discards all etcd event history prior to a given revision. Since etcd uses a multiversion concurrency control model, it preserves all key updates as event history. When the event history up to some revision is no longer needed, all superseded keys may be compacted away to reclaim storage space in the etcd backend database. RPC: Compact #### Options - physical -- 'true' to wait for compaction to physically remove all old revisions #### Output Prints the compacted revision. #### Example ```bash ./etcdctl compaction 1234 # compacted revision 1234 ``` ### WATCH [options] [key or prefix] [range_end] [--] [exec-command arg1 arg2 ...] Watch watches events stream on keys or prefixes, [key or prefix, range_end) if `range-end` is given. The watch command runs until it encounters an error or is terminated by the user. If range_end is given, it must be lexicographically greater than key or "\x00". RPC: Watch #### Options - hex -- print out key and value as hex encode string - interactive -- begins an interactive watch session - prefix -- watch on a prefix if prefix is set. - prev-kv -- get the previous key-value pair before the event happens. - rev -- the revision to start watching. Specifying a revision is useful for observing past events. #### Input format Input is only accepted for interactive mode. ``` watch [options] \n ``` #### Output \[\n\\n\]\n\\n\\n\\n\\n\\n... #### Examples ##### Non-interactive ```bash ./etcdctl watch foo # PUT # foo # bar ``` ```bash ETCDCTL_WATCH_KEY=foo ./etcdctl watch # PUT # foo # bar ``` Receive events and execute `echo watch event received`: ```bash ./etcdctl watch foo -- echo watch event received # PUT # foo # bar # watch event received ``` Watch response is set via `ETCD_WATCH_*` environmental variables: ```bash ./etcdctl watch foo -- sh -c "env | grep ETCD_WATCH_" # PUT # foo # bar # ETCD_WATCH_REVISION=11 # ETCD_WATCH_KEY="foo" # ETCD_WATCH_EVENT_TYPE="PUT" # ETCD_WATCH_VALUE="bar" ``` Watch with environmental variables and execute `echo watch event received`: ```bash export ETCDCTL_WATCH_KEY=foo ./etcdctl watch -- echo watch event received # PUT # foo # bar # watch event received ``` ```bash export ETCDCTL_WATCH_KEY=foo export ETCDCTL_WATCH_RANGE_END=foox ./etcdctl watch -- echo watch event received # PUT # fob # bar # watch event received ``` ##### Interactive ```bash ./etcdctl watch -i watch foo watch foo # PUT # foo # bar # PUT # foo # bar ``` Receive events and execute `echo watch event received`: ```bash ./etcdctl watch -i watch foo -- echo watch event received # PUT # foo # bar # watch event received ``` Watch with environmental variables and execute `echo watch event received`: ```bash export ETCDCTL_WATCH_KEY=foo ./etcdctl watch -i watch -- echo watch event received # PUT # foo # bar # watch event received ``` ```bash export ETCDCTL_WATCH_KEY=foo export ETCDCTL_WATCH_RANGE_END=foox ./etcdctl watch -i watch -- echo watch event received # PUT # fob # bar # watch event received ``` ### LEASE \ LEASE provides commands for key lease management. ### LEASE GRANT \ LEASE GRANT creates a fresh lease with a server-selected time-to-live in seconds greater than or equal to the requested TTL value. RPC: LeaseGrant #### Output Prints a message with the granted lease ID. #### Example ```bash ./etcdctl lease grant 10 # lease 32695410dcc0ca06 granted with TTL(10s) ``` ### LEASE REVOKE \ LEASE REVOKE destroys a given lease, deleting all attached keys. RPC: LeaseRevoke #### Output Prints a message indicating the lease is revoked. #### Example ```bash ./etcdctl lease revoke 32695410dcc0ca06 # lease 32695410dcc0ca06 revoked ``` ### LEASE TIMETOLIVE \ [options] LEASE TIMETOLIVE retrieves the lease information with the given lease ID. RPC: LeaseTimeToLive #### Options - keys -- Get keys attached to this lease #### Output Prints lease information. #### Example ```bash ./etcdctl lease grant 500 # lease 2d8257079fa1bc0c granted with TTL(500s) ./etcdctl put foo1 bar --lease=2d8257079fa1bc0c # OK ./etcdctl put foo2 bar --lease=2d8257079fa1bc0c # OK ./etcdctl lease timetolive 2d8257079fa1bc0c # lease 2d8257079fa1bc0c granted with TTL(500s), remaining(481s) ./etcdctl lease timetolive 2d8257079fa1bc0c --keys # lease 2d8257079fa1bc0c granted with TTL(500s), remaining(472s), attached keys([foo2 foo1]) ./etcdctl lease timetolive 2d8257079fa1bc0c --write-out=json # {"cluster_id":17186838941855831277,"member_id":4845372305070271874,"revision":3,"raft_term":2,"id":3279279168933706764,"ttl":465,"granted-ttl":500,"keys":null} ./etcdctl lease timetolive 2d8257079fa1bc0c --write-out=json --keys # {"cluster_id":17186838941855831277,"member_id":4845372305070271874,"revision":3,"raft_term":2,"id":3279279168933706764,"ttl":459,"granted-ttl":500,"keys":["Zm9vMQ==","Zm9vMg=="]} ./etcdctl lease timetolive 2d8257079fa1bc0c # lease 2d8257079fa1bc0c already expired ``` ### LEASE LIST LEASE LIST lists all active leases. RPC: LeaseLeases #### Output Prints a message with a list of active leases. #### Example ```bash ./etcdctl lease grant 10 # lease 32695410dcc0ca06 granted with TTL(10s) ./etcdctl lease list 32695410dcc0ca06 ``` ### LEASE KEEP-ALIVE \ LEASE KEEP-ALIVE periodically refreshes a lease so it does not expire. RPC: LeaseKeepAlive #### Output Prints a message for every keep alive sent or prints a message indicating the lease is gone. #### Example ```bash ./etcdctl lease keep-alive 32695410dcc0ca0 # lease 32695410dcc0ca0 keepalived with TTL(100) # lease 32695410dcc0ca0 keepalived with TTL(100) # lease 32695410dcc0ca0 keepalived with TTL(100) ... ``` ## Cluster maintenance commands ### MEMBER \ MEMBER provides commands for managing etcd cluster membership. ### MEMBER ADD \ [options] MEMBER ADD introduces a new member into the etcd cluster as a new peer. RPC: MemberAdd #### Options - peer-urls -- comma separated list of URLs to associate with the new member. #### Output Prints the member ID of the new member and the cluster ID. #### Example ```bash ./etcdctl member add newMember --peer-urls=https://127.0.0.1:12345 Member ced000fda4d05edf added to cluster 8c4281cc65c7b112 ETCD_NAME="newMember" ETCD_INITIAL_CLUSTER="newMember=https://127.0.0.1:12345,default=http://10.0.0.30:2380" ETCD_INITIAL_CLUSTER_STATE="existing" ``` ### MEMBER UPDATE \ [options] MEMBER UPDATE sets the peer URLs for an existing member in the etcd cluster. RPC: MemberUpdate #### Options - peer-urls -- comma separated list of URLs to associate with the updated member. #### Output Prints the member ID of the updated member and the cluster ID. #### Example ```bash ./etcdctl member update 2be1eb8f84b7f63e --peer-urls=https://127.0.0.1:11112 # Member 2be1eb8f84b7f63e updated in cluster ef37ad9dc622a7c4 ``` ### MEMBER REMOVE \ MEMBER REMOVE removes a member of an etcd cluster from participating in cluster consensus. RPC: MemberRemove #### Output Prints the member ID of the removed member and the cluster ID. #### Example ```bash ./etcdctl member remove 2be1eb8f84b7f63e # Member 2be1eb8f84b7f63e removed from cluster ef37ad9dc622a7c4 ``` ### MEMBER LIST MEMBER LIST prints the member details for all members associated with an etcd cluster. RPC: [MemberList][member_list_rpc]. #### Output Prints a humanized table of the member IDs, statuses, names, peer addresses, and client addresses. #### Examples ```bash ./etcdctl member list # 8211f1d0f64f3269, started, infra1, http://127.0.0.1:12380, http://127.0.0.1:2379 # 91bc3c398fb3c146, started, infra2, http://127.0.0.1:22380, http://127.0.0.1:22379 # fd422379fda50e48, started, infra3, http://127.0.0.1:32380, http://127.0.0.1:32379 ``` ```bash ./etcdctl -w json member list # {"header":{"cluster_id":17237436991929493444,"member_id":9372538179322589801,"raft_term":2},"members":[{"ID":9372538179322589801,"name":"infra1","peerURLs":["http://127.0.0.1:12380"],"clientURLs":["http://127.0.0.1:2379"]},{"ID":10501334649042878790,"name":"infra2","peerURLs":["http://127.0.0.1:22380"],"clientURLs":["http://127.0.0.1:22379"]},{"ID":18249187646912138824,"name":"infra3","peerURLs":["http://127.0.0.1:32380"],"clientURLs":["http://127.0.0.1:32379"]}]} ``` ```bash ./etcdctl -w table member list +------------------+---------+--------+------------------------+------------------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | +------------------+---------+--------+------------------------+------------------------+ | 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:12380 | http://127.0.0.1:2379 | | 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 | | fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 | +------------------+---------+--------+------------------------+------------------------+ ``` ### ENDPOINT \ ENDPOINT provides commands for querying individual endpoints. #### Options - cluster -- fetch and use all endpoints from the etcd cluster member list ### ENDPOINT HEALTH ENDPOINT HEALTH checks the health of the list of endpoints with respect to cluster. An endpoint is unhealthy when it cannot participate in consensus with the rest of the cluster. #### Output If an endpoint can participate in consensus, prints a message indicating the endpoint is healthy. If an endpoint fails to participate in consensus, prints a message indicating the endpoint is unhealthy. #### Example Check the default endpoint's health: ```bash ./etcdctl endpoint health # 127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.095242ms ``` Check all endpoints for the cluster associated with the default endpoint: ```bash ./etcdctl endpoint --cluster health # http://127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.060091ms # http://127.0.0.1:22379 is healthy: successfully committed proposal: took = 903.138µs # http://127.0.0.1:32379 is healthy: successfully committed proposal: took = 1.113848ms ``` ### ENDPOINT STATUS ENDPOINT STATUS queries the status of each endpoint in the given endpoint list. #### Output ##### Simple format Prints a humanized table of each endpoint URL, ID, version, database size, leadership status, raft term, and raft status. ##### JSON format Prints a line of JSON encoding each endpoint URL, ID, version, database size, leadership status, raft term, and raft status. #### Examples Get the status for the default endpoint: ```bash ./etcdctl endpoint status # 127.0.0.1:2379, 8211f1d0f64f3269, 3.0.0, 25 kB, false, 2, 63 ``` Get the status for the default endpoint as JSON: ```bash ./etcdctl -w json endpoint status # [{"Endpoint":"127.0.0.1:2379","Status":{"header":{"cluster_id":17237436991929493444,"member_id":9372538179322589801,"revision":2,"raft_term":2},"version":"3.0.0","dbSize":24576,"leader":18249187646912138824,"raftIndex":32623,"raftTerm":2}}] ``` Get the status for all endpoints in the cluster associated with the default endpoint: ```bash ./etcdctl -w table endpoint --cluster status +------------------------+------------------+----------------+---------+-----------+-----------+------------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | +------------------------+------------------+----------------+---------+-----------+-----------+------------+ | http://127.0.0.1:2379 | 8211f1d0f64f3269 | 3.2.0-rc.1+git | 25 kB | false | 2 | 8 | | http://127.0.0.1:22379 | 91bc3c398fb3c146 | 3.2.0-rc.1+git | 25 kB | false | 2 | 8 | | http://127.0.0.1:32379 | fd422379fda50e48 | 3.2.0-rc.1+git | 25 kB | true | 2 | 8 | +------------------------+------------------+----------------+---------+-----------+-----------+------------+ ``` ### ENDPOINT HASHKV ENDPOINT HASHKV fetches the hash of the key-value store of an endpoint. #### Output ##### Simple format Prints a humanized table of each endpoint URL and KV history hash. ##### JSON format Prints a line of JSON encoding each endpoint URL and KV history hash. #### Examples Get the hash for the default endpoint: ```bash ./etcdctl endpoint hashkv # 127.0.0.1:2379, 1084519789 ``` Get the status for the default endpoint as JSON: ```bash ./etcdctl -w json endpoint hashkv # [{"Endpoint":"127.0.0.1:2379","Hash":{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":1,"raft_term":3},"hash":1084519789,"compact_revision":-1}}] ``` Get the status for all endpoints in the cluster associated with the default endpoint: ```bash ./etcdctl -w table endpoint --cluster hashkv +------------------------+------------+ | ENDPOINT | HASH | +------------------------+------------+ | http://127.0.0.1:12379 | 1084519789 | | http://127.0.0.1:22379 | 1084519789 | | http://127.0.0.1:32379 | 1084519789 | +------------------------+------------+ ``` ### ALARM \ Provides alarm related commands ### ALARM DISARM `alarm disarm` Disarms all alarms RPC: Alarm #### Output `alarm:` if alarm is present and disarmed. #### Examples ```bash ./etcdctl alarm disarm ``` If NOSPACE alarm is present: ```bash ./etcdctl alarm disarm # alarm:NOSPACE ``` ### ALARM LIST `alarm list` lists all alarms. RPC: Alarm #### Output `alarm:` if alarm is present, empty string if no alarms present. #### Examples ```bash ./etcdctl alarm list ``` If NOSPACE alarm is present: ```bash ./etcdctl alarm list # alarm:NOSPACE ``` ### DEFRAG [options] DEFRAG defragments the backend database file for a set of given endpoints while etcd is running, or directly defragments an etcd data directory while etcd is not running. When an etcd member reclaims storage space from deleted and compacted keys, the space is kept in a free list and the database file remains the same size. By defragmenting the database, the etcd member releases this free space back to the file system. **Note that defragmentation to a live member blocks the system from reading and writing data while rebuilding its states.** **Note that defragmentation request does not get replicated over cluster. That is, the request is only applied to the local node. Specify all members in `--endpoints` flag.** #### Options - data-dir -- Optional. If present, defragments a data directory not in use by etcd. #### Output For each endpoints, prints a message indicating whether the endpoint was successfully defragmented. #### Example ```bash ./etcdctl --endpoints=localhost:2379,badendpoint:2379 defrag # Finished defragmenting etcd member[localhost:2379] # Failed to defragment etcd member[badendpoint:2379] (grpc: timed out trying to connect) ``` To defragment a data directory directly, use the `--data-dir` flag: ``` bash # Defragment while etcd is not running ./etcdctl defrag --data-dir default.etcd # success (exit status 0) # Error: cannot open database at default.etcd/member/snap/db ``` #### Remarks DEFRAG returns a zero exit code only if it succeeded defragmenting all given endpoints. ### SNAPSHOT \ SNAPSHOT provides commands to restore a snapshot of a running etcd server into a fresh cluster. ### SNAPSHOT SAVE \ SNAPSHOT SAVE writes a point-in-time snapshot of the etcd backend database to a file. #### Output The backend snapshot is written to the given file path. #### Example Save a snapshot to "snapshot.db": ``` ./etcdctl snapshot save snapshot.db ``` ### SNAPSHOT RESTORE [options] \ SNAPSHOT RESTORE creates an etcd data directory for an etcd cluster member from a backend database snapshot and a new cluster configuration. Restoring the snapshot into each member for a new cluster configuration will initialize a new etcd cluster preloaded by the snapshot data. #### Options The snapshot restore options closely resemble to those used in the `etcd` command for defining a cluster. - data-dir -- Path to the data directory. Uses \.etcd if none given. - wal-dir -- Path to the WAL directory. Uses data directory if none given. - initial-cluster -- The initial cluster configuration for the restored etcd cluster. - initial-cluster-token -- Initial cluster token for the restored etcd cluster. - initial-advertise-peer-urls -- List of peer URLs for the member being restored. - name -- Human-readable name for the etcd cluster member being restored. - skip-hash-check -- Ignore snapshot integrity hash value (required if copied from data directory) #### Output A new etcd data directory initialized with the snapshot. #### Example Save a snapshot, restore into a new 3 node cluster, and start the cluster: ``` ./etcdctl snapshot save snapshot.db # restore members bin/etcdctl snapshot restore snapshot.db --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://127.0.0.1:12380 --name sshot1 --initial-cluster 'sshot1=http://127.0.0.1:12380,sshot2=http://127.0.0.1:22380,sshot3=http://127.0.0.1:32380' bin/etcdctl snapshot restore snapshot.db --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://127.0.0.1:22380 --name sshot2 --initial-cluster 'sshot1=http://127.0.0.1:12380,sshot2=http://127.0.0.1:22380,sshot3=http://127.0.0.1:32380' bin/etcdctl snapshot restore snapshot.db --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://127.0.0.1:32380 --name sshot3 --initial-cluster 'sshot1=http://127.0.0.1:12380,sshot2=http://127.0.0.1:22380,sshot3=http://127.0.0.1:32380' # launch members bin/etcd --name sshot1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 & bin/etcd --name sshot2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 & bin/etcd --name sshot3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 & ``` ### SNAPSHOT STATUS \ SNAPSHOT STATUS lists information about a given backend database snapshot file. #### Output ##### Simple format Prints a humanized table of the database hash, revision, total keys, and size. ##### JSON format Prints a line of JSON encoding the database hash, revision, total keys, and size. #### Examples ```bash ./etcdctl snapshot status file.db # cf1550fb, 3, 3, 25 kB ``` ```bash ./etcdctl -write-out=json snapshot status file.db # {"hash":3474280699,"revision":3,"totalKey":3,"totalSize":24576} ``` ```bash ./etcdctl -write-out=table snapshot status file.db +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ | cf1550fb | 3 | 3 | 25 kB | +----------+----------+------------+------------+ ``` ### MOVE-LEADER \ MOVE-LEADER transfers leadership from the leader to another member in the cluster. #### Example ```bash # to choose transferee transferee_id=$(./etcdctl \ --endpoints localhost:12379,localhost:22379,localhost:32379 \ endpoint status | grep -m 1 "false" | awk -F', ' '{print $2}') echo ${transferee_id} # c89feb932daef420 # endpoints should include leader node ./etcdctl --endpoints ${transferee_ep} move-leader ${transferee_id} # Error: no leader endpoint given at [localhost:22379 localhost:32379] # request to leader with target node ID ./etcdctl --endpoints ${leader_ep} move-leader ${transferee_id} # Leadership transferred from 45ddc0e800e20b93 to c89feb932daef420 ``` ## Concurrency commands ### LOCK [options] \ [command arg1 arg2 ...] LOCK acquires a distributed named mutex with a given name. Once the lock is acquired, it will be held until etcdctl is terminated. #### Options - ttl - time out in seconds of lock session. #### Output Once the lock is acquired, the result for the GET on the unique lock holder key is displayed. If a command is given, it will be launched with environment variables `ETCD_LOCK_KEY` and `ETCD_LOCK_REV` set to the lock's holder key and revision. #### Example Acquire lock with standard output display: ```bash ./etcdctl lock mylock # mylock/1234534535445 ``` Acquire lock and execute `echo lock acquired`: ```bash ./etcdctl lock mylock echo lock acquired # lock acquired ``` #### Remarks LOCK returns a zero exit code only if it is terminated by a signal and releases the lock. If LOCK is abnormally terminated or fails to contact the cluster to release the lock, the lock will remain held until the lease expires. Progress may be delayed by up to the default lease length of 60 seconds. ### ELECT [options] \ [proposal] ELECT participates on a named election. A node announces its candidacy in the election by providing a proposal value. If a node wishes to observe the election, ELECT listens for new leaders values. Whenever a leader is elected, its proposal is given as output. #### Options - listen -- observe the election. #### Output - If a candidate, ELECT displays the GET on the leader key once the node is elected election. - If observing, ELECT streams the result for a GET on the leader key for the current election and all future elections. #### Example ```bash ./etcdctl elect myelection foo # myelection/1456952310051373265 # foo ``` #### Remarks ELECT returns a zero exit code only if it is terminated by a signal and can revoke its candidacy or leadership, if any. If a candidate is abnormally terminated, election rogress may be delayed by up to the default lease length of 60 seconds. ## Authentication commands ### AUTH \ `auth enable` activates authentication on an etcd cluster and `auth disable` deactivates. When authentication is enabled, etcd checks all requests for appropriate authorization. RPC: AuthEnable/AuthDisable #### Output `Authentication Enabled`. #### Examples ```bash ./etcdctl user add root # Password of root:#type password for root # Type password of root again for confirmation:#re-type password for root # User root created ./etcdctl user grant-role root root # Role root is granted to user root ./etcdctl user get root # User: root # Roles: root ./etcdctl role add root # Role root created ./etcdctl role get root # Role root # KV Read: # KV Write: ./etcdctl auth enable # Authentication Enabled ``` ### ROLE \ ROLE is used to specify differnt roles which can be assigned to etcd user(s). ### ROLE ADD \ `role add` creates a role. RPC: RoleAdd #### Output `Role created`. #### Examples ```bash ./etcdctl --user=root:123 role add myrole # Role myrole created ``` ### ROLE GET \ `role get` lists detailed role information. RPC: RoleGet #### Output Detailed role information. #### Examples ```bash ./etcdctl --user=root:123 role get myrole # Role myrole # KV Read: # foo # KV Write: # foo ``` ### ROLE DELETE \ `role delete` deletes a role. RPC: RoleDelete #### Output `Role deleted`. #### Examples ```bash ./etcdctl --user=root:123 role delete myrole # Role myrole deleted ``` ### ROLE LIST \ `role list` lists all roles in etcd. RPC: RoleList #### Output A role per line. #### Examples ```bash ./etcdctl --user=root:123 role list # roleA # roleB # myrole ``` ### ROLE GRANT-PERMISSION [options] \ \ \ [endkey] `role grant-permission` grants a key to a role. RPC: RoleGrantPermission #### Options - from-key -- grant a permission of keys that are greater than or equal to the given key using byte compare - prefix -- grant a prefix permission #### Output `Role updated`. #### Examples Grant read and write permission on the key `foo` to role `myrole`: ```bash ./etcdctl --user=root:123 role grant-permission myrole readwrite foo # Role myrole updated ``` Grant read permission on the wildcard key pattern `foo/*` to role `myrole`: ```bash ./etcdctl --user=root:123 role grant-permission --prefix myrole readwrite foo/ # Role myrole updated ``` ### ROLE REVOKE-PERMISSION \ \ \ [endkey] `role revoke-permission` revokes a key from a role. RPC: RoleRevokePermission #### Options - from-key -- revoke a permission of keys that are greater than or equal to the given key using byte compare - prefix -- revoke a prefix permission #### Output `Permission of key is revoked from role ` for single key. `Permission of range [, ) is revoked from role ` for a key range. Exit code is zero. #### Examples ```bash ./etcdctl --user=root:123 role revoke-permission myrole foo # Permission of key foo is revoked from role myrole ``` ### USER \ USER provides commands for managing users of etcd. ### USER ADD \ [options] `user add` creates a user. RPC: UserAdd #### Options - interactive -- Read password from stdin instead of interactive terminal #### Output `User created`. #### Examples ```bash ./etcdctl --user=root:123 user add myuser # Password of myuser: #type password for my user # Type password of myuser again for confirmation:#re-type password for my user # User myuser created ``` ### USER GET \ [options] `user get` lists detailed user information. RPC: UserGet #### Options - detail -- Show permissions of roles granted to the user #### Output Detailed user information. #### Examples ```bash ./etcdctl --user=root:123 user get myuser # User: myuser # Roles: ``` ### USER DELETE \ `user delete` deletes a user. RPC: UserDelete #### Output `User deleted`. #### Examples ```bash ./etcdctl --user=root:123 user delete myuser # User myuser deleted ``` ### USER LIST `user list` lists detailed user information. RPC: UserList #### Output - List of users, one per line. #### Examples ```bash ./etcdctl --user=root:123 user list # user1 # user2 # myuser ``` ### USER PASSWD \ [options] `user passwd` changes a user's password. RPC: UserChangePassword #### Options - interactive -- if true, read password in interactive terminal #### Output `Password updated`. #### Examples ```bash ./etcdctl --user=root:123 user passwd myuser # Password of myuser: #type new password for my user # Type password of myuser again for confirmation: #re-type the new password for my user # Password updated ``` ### USER GRANT-ROLE \ \ `user grant-role` grants a role to a user RPC: UserGrantRole #### Output `Role is granted to user `. #### Examples ```bash ./etcdctl --user=root:123 user grant-role userA roleA # Role roleA is granted to user userA ``` ### USER REVOKE-ROLE \ \ `user revoke-role` revokes a role from a user RPC: UserRevokeRole #### Output `Role is revoked from user `. #### Examples ```bash ./etcdctl --user=root:123 user revoke-role userA roleA # Role roleA is revoked from user userA ``` ## Utility commands ### MAKE-MIRROR [options] \ [make-mirror][mirror] mirrors a key prefix in an etcd cluster to a destination etcd cluster. #### Options - dest-cacert -- TLS certificate authority file for destination cluster - dest-cert -- TLS certificate file for destination cluster - dest-key -- TLS key file for destination cluster - prefix -- The key-value prefix to mirror - dest-prefix -- The destination prefix to mirror a prefix to a different prefix in the destination cluster - no-dest-prefix -- Mirror key-values to the root of the destination cluster - dest-insecure-transport -- Disable transport security for client connections #### Output The approximate total number of keys transferred to the destination cluster, updated every 30 seconds. #### Examples ``` ./etcdctl make-mirror mirror.example.com:2379 # 10 # 18 ``` [mirror]: ./doc/mirror_maker.md ### MIGRATE [options] Migrates keys in a v2 store to a v3 mvcc store. Users should run migration command for all members in the cluster. #### Options - data-dir -- Path to the data directory - wal-dir -- Path to the WAL directory - transformer -- Path to the user-provided transformer program (default if not provided) #### Output No output on success. #### Default transformer If user does not provide a transformer program, migrate command will use the default transformer. The default transformer transforms `storev2` formatted keys into `mvcc` formatted keys according to the following Go program: ```go func transform(n *storev2.Node) *mvccpb.KeyValue { if n.Dir { return nil } kv := &mvccpb.KeyValue{ Key: []byte(n.Key), Value: []byte(n.Value), CreateRevision: int64(n.CreatedIndex), ModRevision: int64(n.ModifiedIndex), Version: 1, } return kv } ``` #### User-provided transformer Users can provide a customized 1:n transformer function that transforms a key from the v2 store to any number of keys in the mvcc store. The migration program writes JSON formatted [v2 store keys][v2key] to the transformer program's stdin, reads protobuf formatted [mvcc keys][v3key] back from the transformer program's stdout, and finishes migration by saving the transformed keys into the mvcc store. The provided transformer should read until EOF and flush the stdout before exiting to ensure data integrity. #### Example ``` ./etcdctl migrate --data-dir=/var/etcd --transformer=k8s-transformer # finished transforming keys ``` ### VERSION Prints the version of etcdctl. #### Output Prints etcd version and API version. #### Examples ```bash ./etcdctl version # etcdctl version: 3.1.0-alpha.0+git # API version: 3.1 ``` ## Exit codes For all commands, a successful execution return a zero exit code. All failures will return non-zero exit codes. ## Output formats All commands accept an output format by setting `-w` or `--write-out`. All commands default to the "simple" output format, which is meant to be human-readable. The simple format is listed in each command's `Output` description since it is customized for each command. If a command has a corresponding RPC, it will respect all output formats. If a command fails, returning a non-zero exit code, an error string will be written to standard error regardless of output format. ### Simple A format meant to be easy to parse and human-readable. Specific to each command. ### JSON The JSON encoding of the command's [RPC response][etcdrpc]. Since etcd's RPCs use byte strings, the JSON output will encode keys and values in base64. Some commands without an RPC also support JSON; see the command's `Output` description. ### Protobuf The protobuf encoding of the command's [RPC response][etcdrpc]. If an RPC is streaming, the stream messages will be concetenated. If an RPC is not given for a command, the protobuf output is not defined. ### Fields An output format similar to JSON but meant to parse with coreutils. For an integer field named `Field`, it writes a line in the format `"Field" : %d` where `%d` is go's integer formatting. For byte array fields, it writes `"Field" : %q` where `%q` is go's quoted string formatting (e.g., `[]byte{'a', '\n'}` is written as `"a\n"`). ## Compatibility Support etcdctl is still in its early stage. We try out best to ensure fully compatible releases, however we might break compatibility to fix bugs or improve commands. If we intend to release a version of etcdctl with backward incompatibilities, we will provide notice prior to release and have instructions on how to upgrade. ### Input Compatibility Input includes the command name, its flags, and its arguments. We ensure backward compatibility of the input of normal commands in non-interactive mode. ### Output Compatibility Output includes output from etcdctl and its exit code. etcdctl provides `simple` output format by default. We ensure compatibility for the `simple` output format of normal commands in non-interactive mode. Currently, we do not ensure backward compatibility for `JSON` format and the format in non-interactive mode. Currently, we do not ensure backward compatibility of utility commands. ### TODO: compatibility with etcd server [etcd]: https://github.com/coreos/etcd [READMEv2]: READMEv2.md [v2key]: ../store/node_extern.go#L28-L37 [v3key]: ../mvcc/mvccpb/kv.proto#L12-L29 [etcdrpc]: ../etcdserver/etcdserverpb/rpc.proto [storagerpc]: ../mvcc/mvccpb/kv.proto [member_list_rpc]: ../etcdserver/etcdserverpb/rpc.proto#L493-L497 --- ### Overview (`third_party/coreos-etcd/functional/README.md`) ## etcd Functional Testing [`functional`](https://godoc.org/github.com/coreos/etcd/functional) verifies the correct behavior of etcd under various system and network malfunctions. It sets up an etcd cluster under high pressure loads and continuously injects failures into the cluster. Then it expects the etcd cluster to recover within a few seconds. This has been extremely helpful to find critical bugs. See [`rpcpb.Case`](https://godoc.org/github.com/coreos/etcd/functional/rpcpb#Case) for all failure cases. See [functional.yaml](https://github.com/coreos/etcd/blob/master/functional.yaml) for an example configuration. ### Run locally ```bash PASSES=functional ./test ``` ### Run with Docker ```bash pushd .. make build-docker-functional popd ``` And run [example scripts](./scripts). ```bash # run 3 agents for 3-node local etcd cluster ./scripts/docker-local-agent.sh 1 ./scripts/docker-local-agent.sh 2 ./scripts/docker-local-agent.sh 3 # to run only 1 tester round ./scripts/docker-local-tester.sh ``` ## etcd Proxy Proxy layer that simulates various network conditions. Test locally ```bash $ ./build $ ./bin/etcd $ make build-functional $ ./bin/etcd-proxy --help $ ./bin/etcd-proxy --from localhost:23790 --to localhost:2379 --http-port 2378 --verbose $ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:2379 put foo bar $ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:23790 put foo bar ``` Proxy overhead per request is under 500μs ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Delay client transmit ```bash $ curl -L http://localhost:2378/delay-tx -X PUT \ -d "latency=5s&random-variable=100ms" # added send latency 5s±100ms (current latency 4.92143955s) $ curl -L http://localhost:2378/delay-tx # current send latency 4.92143955s $ ETCDCTL_API=3 ./bin/etcdctl \ --endpoints localhost:23790 \ --command-timeout=3s \ put foo bar # Error: context deadline exceeded $ curl -L http://localhost:2378/delay-tx -X DELETE # removed latency 4.92143955s $ curl -L http://localhost:2378/delay-tx # current send latency 0s $ ETCDCTL_API=3 ./bin/etcdctl \ --endpoints localhost:23790 \ --command-timeout=3s \ put foo bar # OK ``` Pause client transmit ```bash $ curl -L http://localhost:2378/pause-tx -X PUT # paused forwarding [tcp://localhost:23790 -> tcp://localhost:2379] $ ETCDCTL_API=3 ./bin/etcdctl \ --endpoints localhost:23790 \ put foo bar # Error: context deadline exceeded $ curl -L http://localhost:2378/pause-tx -X DELETE # unpaused forwarding [tcp://localhost:23790 -> tcp://localhost:2379] ``` Drop client packets ```bash $ curl -L http://localhost:2378/blackhole-tx -X PUT # blackholed; dropping packets [tcp://localhost:23790 -> tcp://localhost:2379] $ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:23790 put foo bar # Error: context deadline exceeded $ curl -L http://localhost:2378/blackhole-tx -X DELETE # unblackholed; restart forwarding [tcp://localhost:23790 -> tcp://localhost:2379] ``` Trigger leader election ```bash $ ./build $ make build-functional $ rm -rf /tmp/etcd-proxy-data.s* $ goreman -f ./functional/Procfile-proxy start $ ETCDCTL_API=3 ./bin/etcdctl \ --endpoints localhost:13790,localhost:23790,localhost:33790 \ member list # isolate s1 when s1 is the current leader $ curl -L http://localhost:1381/blackhole-tx -X PUT $ curl -L http://localhost:1381/blackhole-rx -X PUT # s1 becomes follower after election timeout ``` --- ### Overview (`third_party/coreos-etcd/pkg/README.md`) pkg/ is a collection of utility packages used by etcd without being specific to etcd itself. A package belongs here only if it could possibly be moved out into its own repository in the future. --- ### Overview (`third_party/coreos-etcd/raft/README.md`) # Raft library Raft is a protocol with which a cluster of nodes can maintain a replicated state machine. The state machine is kept in sync through the use of a replicated log. For more details on Raft, see "In Search of an Understandable Consensus Algorithm" (https://ramcloud.stanford.edu/raft.pdf) by Diego Ongaro and John Ousterhout. This Raft library is stable and feature complete. As of 2016, it is **the most widely used** Raft library in production, serving tens of thousands clusters each day. It powers distributed systems such as etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, and more. Most Raft implementations have a monolithic design, including storage handling, messaging serialization, and network transport. This library instead follows a minimalistic design philosophy by only implementing the core raft algorithm. This minimalism buys flexibility, determinism, and performance. To keep the codebase small as well as provide flexibility, the library only implements the Raft algorithm; both network and disk IO are left to the user. Library users must implement their own transportation layer for message passing between Raft peers over the wire. Similarly, users must implement their own storage layer to persist the Raft log and state. In order to easily test the Raft library, its behavior should be deterministic. To achieve this determinism, the library models Raft as a state machine. The state machine takes a `Message` as input. A message can either be a local timer update or a network message sent from a remote peer. The state machine's output is a 3-tuple `{[]Messages, []LogEntries, NextState}` consisting of an array of `Messages`, `log entries`, and `Raft state changes`. For state machines with the same state, the same state machine input should always generate the same state machine output. A simple example application, _raftexample_, is also available to help illustrate how to use this package in practice: https://github.com/coreos/etcd/tree/master/contrib/raftexample # Features This raft implementation is a full feature implementation of Raft protocol. Features includes: - Leader election - Log replication - Log compaction - Membership changes - Leadership transfer extension - Efficient linearizable read-only queries served by both the leader and followers - leader checks with quorum and bypasses Raft log before processing read-only queries - followers asks leader to get a safe read index before processing read-only queries - More efficient lease-based linearizable read-only queries served by both the leader and followers - leader bypasses Raft log and processing read-only queries locally - followers asks leader to get a safe read index before processing read-only queries - this approach relies on the clock of the all the machines in raft group This raft implementation also includes a few optional enhancements: - Optimistic pipelining to reduce log replication latency - Flow control for log replication - Batching Raft messages to reduce synchronized network I/O calls - Batching log entries to reduce disk synchronized I/O - Writing to leader's disk in parallel - Internal proposal redirection from followers to leader - Automatic stepping down when the leader loses quorum ## Notable Users - [cockroachdb](https://github.com/cockroachdb/cockroach) A Scalable, Survivable, Strongly-Consistent SQL Database - [dgraph](https://github.com/dgraph-io/dgraph) A Scalable, Distributed, Low Latency, High Throughput Graph Database - [etcd](https://github.com/coreos/etcd) A distributed reliable key-value store - [tikv](https://github.com/pingcap/tikv) A Distributed transactional key value database powered by Rust and Raft - [swarmkit](https://github.com/docker/swarmkit) A toolkit for orchestrating distributed systems at any scale. - [chain core](https://github.com/chain/chain) Software for operating permissioned, multi-asset blockchain networks ## Usage The primary object in raft is a Node. Either start a Node from scratch using raft.StartNode or start a Node from some initial state using raft.RestartNode. To start a three-node cluster ```go storage := raft.NewMemoryStorage() c := &Config{ ID: 0x01, ElectionTick: 10, HeartbeatTick: 1, Storage: storage, MaxSizePerMsg: 4096, MaxInflightMsgs: 256, } // Set peer list to the other nodes in the cluster. // Note that they need to be started separately as well. n := raft.StartNode(c, []raft.Peer{{ID: 0x02}, {ID: 0x03}}) ``` Start a single node cluster, like so: ```go // Create storage and config as shown above. // Set peer list to itself, so this node can become the leader of this single-node cluster. peers := []raft.Peer{{ID: 0x01}} n := raft.StartNode(c, peers) ``` To allow a new node to join this cluster, do not pass in any peers. First, add the node to the existing cluster by calling `ProposeConfChange` on any existing node inside the cluster. Then, start the node with an empty peer list, like so: ```go // Create storage and config as shown above. n := raft.StartNode(c, nil) ``` To restart a node from previous state: ```go storage := raft.NewMemoryStorage() // Recover the in-memory storage from persistent snapshot, state and entries. storage.ApplySnapshot(snapshot) storage.SetHardState(state) storage.Append(entries) c := &Config{ ID: 0x01, ElectionTick: 10, HeartbeatTick: 1, Storage: storage, MaxSizePerMsg: 4096, MaxInflightMsgs: 256, } // Restart raft without peer information. // Peer information is already included in the storage. n := raft.RestartNode(c) ``` After creating a Node, the user has a few responsibilities: First, read from the Node.Ready() channel and process the updates it contains. These steps may be performed in parallel, except as noted in step 2. 1. Write Entries, HardState and Snapshot to persistent storage in order, i.e. Entries first, then HardState and Snapshot if they are not empty. If persistent storage supports atomic writes then all of them can be written together. Note that when writing an Entry with Index i, any previously-persisted entries with Index >= i must be discarded. 2. Send all Messages to the nodes named in the To field. It is important that no messages be sent until the latest HardState has been persisted to disk, and all Entries written by any previous Ready batch (Messages may be sent while entries from the same batch are being persisted). To reduce the I/O latency, an optimization can be applied to make leader write to disk in parallel with its followers (as explained at section 10.2.1 in Raft thesis). If any Message has type MsgSnap, call Node.ReportSnapshot() after it has been sent (these messages may be large). Note: Marshalling messages is not thread-safe; it is important to make sure that no new entries are persisted while marshalling. The easiest way to achieve this is to serialise the messages directly inside the main raft loop. 3. Apply Snapshot (if any) and CommittedEntries to the state machine. If any committed Entry has Type EntryConfChange, call Node.ApplyConfChange() to apply it to the node. The configuration change may be cancelled at this point by setting the NodeID field to zero before calling ApplyConfChange (but ApplyConfChange must be called one way or the other, and the decision to cancel must be based solely on the state machine and not external information such as the observed health of the node). 4. Call Node.Advance() to signal readiness for the next batch of updates. This may be done at any time after step 1, although all updates must be processed in the order they were returned by Ready. Second, all persisted log entries must be made available via an implementation of the Storage interface. The provided MemoryStorage type can be used for this (if repopulating its state upon a restart), or a custom disk-backed implementation can be supplied. Third, after receiving a message from another node, pass it to Node.Step: ```go func recvRaftRPC(ctx context.Context, m raftpb.Message) { n.Step(ctx, m) } ``` Finally, call `Node.Tick()` at regular intervals (probably via a `time.Ticker`). Raft has two important timeouts: heartbeat and the election timeout. However, internally to the raft package time is represented by an abstract "tick". The total state machine handling loop will look something like this: ```go for { select { case <-s.Ticker: n.Tick() case rd := <-s.Node.Ready(): saveToStorage(rd.State, rd.Entries, rd.Snapshot) send(rd.Messages) if !raft.IsEmptySnap(rd.Snapshot) { processSnapshot(rd.Snapshot) } for _, entry := range rd.CommittedEntries { process(entry) if entry.Type == raftpb.EntryConfChange { var cc raftpb.ConfChange cc.Unmarshal(entry.Data) s.Node.ApplyConfChange(cc) } } s.Node.Advance() case <-s.done: return } } ``` To propose changes to the state machine from the node to take application data, serialize it into a byte slice and call: ```go n.Propose(ctx, data) ``` If the proposal is committed, data will appear in committed entries with type raftpb.EntryNormal. There is no guarantee that a proposed command will be committed; the command may have to be reproposed after a timeout. To add or remove node in a cluster, build ConfChange struct 'cc' and call: ```go n.ProposeConfChange(ctx, cc) ``` After config change is committed, some committed entry with type raftpb.EntryConfChange will be returned. This must be applied to node through: ```go var cc raftpb.ConfChange cc.Unmarshal(data) n.ApplyConfChange(cc) ``` Note: An ID represents a unique node in a cluster for all time. A given ID MUST be used only once even if the old node has been removed. This means that for example IP addresses make poor node IDs since they may be reused. Node IDs must be non-zero. ## Implementation notes This implementation is up to date with the final Raft thesis (https://ramcloud.stanford.edu/~ongaro/thesis.pdf), although this implementation of the membership change protocol differs somewhat from that described in chapter 4. The key invariant that membership changes happen one node at a time is preserved, but in our implementation the membership change takes effect when its entry is applied, not when it is added to the log (so the entry is committed under the old membership instead of the new). This is equivalent in terms of safety, since the old and new configurations are guaranteed to overlap. To ensure there is no attempt to commit two membership changes at once by matching log positions (which would be unsafe since they should have different quorum requirements), any proposed membership change is simply disallowed while any uncommitted change appears in the leader's log. This approach introduces a problem when removing a member from a two-member cluster: If one of the members dies before the other one receives the commit of the confchange entry, then the member cannot be removed any more since the cluster cannot make progress. For this reason it is highly recommended to use three or more nodes in every cluster. --- ### Overview (`hack/contrib/docker/grctl/README.md`) ## Install ### necessary condition `~/.kube/config` exist ### install shell ```bash docker run -it --rm -v /:/rootfs goodrain.me/rbd-grctl:V5.2-dev copy grctl install ``` --- ### Overview (`hack/contrib/docker/registry-gc/README.md`) # Registry coordination image This image contains the coordinator, the one-shot GC executor, and the native Registry binary from the explicitly supplied image. Build it from the same Core revision as `rbd-api` and publish it outside the Registry it will coordinate. ```sh docker build \ --build-arg REGISTRY_IMAGE="$REGISTRY_NATIVE_IMAGE" \ -f hack/contrib/docker/registry-gc/Dockerfile \ -t "$GC_COORDINATOR_IMAGE" . ``` `REGISTRY_NATIVE_IMAGE` must include the digest observed on the target cluster. The build rejects mutable tags. Use `/registry-coordinator` as the sidecar command; the same image supplies `/registry-gc` for server-generated GC Jobs. ## Platform-scoped control authentication Set these coordinator arguments when using the Console bridge: ```text --coordination-api= --console-enterprise= --console-region= --console-system-identity=true --credential-file=/control/key --permit-key-file=/permit/key ``` The control key is platform-owned and independent of the plugin gateway credential. It must survive plugin uninstall/reinstall. The independent permit key is shared only by `rbd-api` and the coordinator. Configure the API with `CLEANUP_REGISTRY_PERMIT_KEY_FILE=/var/run/cleanup-registry-permit/key` and project that Secret read-only into both components. Do not reuse the control key as the permit key. No Region administrator token or private client certificate is needed in signed Console mode. Plain HTTP requires explicit `--allow-internal-http=true` for the trusted internal Console origin. Console must include the system coordination endpoint and mount a platform-owned Secret entry at the absolute path configured by `CLEANUP_SYSTEM_COORDINATION_FILE`. That entry is one JSON document containing exactly `enterprise`, `region`, and `key` (base64 encoding of the control key bytes). The enterprise and region must match both the client scope and a current Console enterprise-region association. The coordinator mounts the same control key bytes as `/control/key`; it does not mount the JSON document. Use a generated printable key compatible with the CLI credential-file parser. Never place values in application templates, logs or chat. Missing or invalid configuration fails closed; there is no plugin-key fallback. The full system URL is part of the signature, preventing replay to the plugin URL. The default client mode remains the installation endpoint for existing callers. Operator-managed Registry coordination explicitly enables system identity; GC Jobs preserve that flag. Do not rotate a key underneath active GC without a coordinated drain/restart: Console reloads the file, while clients hold the key loaded at startup. Provisioning and controlled deactivation remain rollout gates. Use separate Secrets and volumes for the two keys. GC Jobs inherit the Console scope and control credential from the verified coordinator Pod, but never receive the permit signing Secret. The launcher rejects shared Secret projections and incomplete scopes. Invalid or missing explicit permit files fail closed, including during rotation. Legacy direct-token mode remains supported when Console scope arguments are absent. Storage IDs, generations and physical volume identities must come from Core's verified registry preparation response. Enrollment in `collecting` mode does not authorize deletion. Storage, ingress, producer coverage, references, and original execution identity still have to pass the existing readiness checks. ## Deployment boundary These binaries and authentication settings are prerequisites. Persistent Operator support for the coordinator container and coordinated Service/probe configuration must be installed before switching a managed `rbd-hub`. A manual Deployment edit is not a durable installation method. Live ingress changes require the separate review agreed for this rollout. GC remains an explicitly confirmed, one-shot job. ### Initialize independent platform credentials The installer helper creates only `rbd-system/rainbond-registry-control` and `rbd-system/rainbond-registry-permit`. It does not patch workloads or ingress. Select the target Kubernetes context explicitly: ```sh python3 hack/contrib/docker/registry-gc/provision_identity.py \ --context --enterprise --region ``` Existing Secrets must have matching scope and purpose, no plugin ownerReference, consistent Console projection, and independent keys. They are never overwritten. A partial creation can be retried after checking state; a conflicting identity requires investigation. The helper prints no credential data or kubectl errors. Secrets are immutable to prevent accidental in-place rotation during active work. Rotation requires separate versioned Secrets and a reviewed drain/migration. The control Secret supplies `key` to the coordinator and `console.json` to Console. The permit Secret supplies `key` only to Core API and the coordinator. Do not project the entire control Secret into a plugin or permit Secret into GC Jobs. ### Console runtime identity The Console image must contain the private writer-announcement client. Configure these non-sensitive downward-API entries on `rbd-app-ui` (RbdComponent `spec.env`): ```yaml - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid ``` After a valid signed inventory request and enterprise/region ownership check, Console announces its own Pod name/UID and the supported reference protocol using its existing verified Region TLS client. Core observes the actual container and image identity. No caller-provided image or writable flag is accepted. The registration path is deliberately absent from the plugin/system bridge allowlist. Missing identity or an older Core does not break read-only inventory, but supplies no writer evidence. Success is cached per Console process and region; the record is not a readiness grant. Completeness checks must still account for every live Console instance; one successful request does not certify other replicas. --- ### Overview (`third_party/coreos-etcd/contrib/raftexample/README.md`) # raftexample raftexample is an example usage of etcd's [raft library](../../raft). It provides a simple REST API for a key-value store cluster backed by the [Raft][raft] consensus algorithm. [raft]: http://raftconsensus.github.io/ ## Getting Started ### Running single node raftexample First start a single-member cluster of raftexample: ```sh raftexample --id 1 --cluster http://127.0.0.1:12379 --port 12380 ``` Each raftexample process maintains a single raft instance and a key-value server. The process's list of comma separated peers (--cluster), its raft ID index into the peer list (--id), and http key-value server port (--port) are passed through the command line. Next, store a value ("hello") to a key ("my-key"): ``` curl -L http://127.0.0.1:12380/my-key -XPUT -d hello ``` Finally, retrieve the stored key: ``` curl -L http://127.0.0.1:12380/my-key ``` ### Running a local cluster First install [goreman](https://github.com/mattn/goreman), which manages Procfile-based applications. The [Procfile script](./Procfile) will set up a local example cluster. Start it with: ```sh goreman start ``` This will bring up three raftexample instances. Now it's possible to write a key-value pair to any member of the cluster and likewise retrieve it from any member. ### Fault Tolerance To test cluster recovery, first start a cluster and write a value "foo": ```sh goreman start curl -L http://127.0.0.1:12380/my-key -XPUT -d foo ``` Next, remove a node and replace the value with "bar" to check cluster availability: ```sh goreman run stop raftexample2 curl -L http://127.0.0.1:12380/my-key -XPUT -d bar curl -L http://127.0.0.1:32380/my-key ``` Finally, bring the node back up and verify it recovers with the updated value "bar": ```sh goreman run start raftexample2 curl -L http://127.0.0.1:22380/my-key ``` ### Dynamic cluster reconfiguration Nodes can be added to or removed from a running cluster using requests to the REST API. For example, suppose we have a 3-node cluster that was started with the commands: ```sh raftexample --id 1 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 --port 12380 raftexample --id 2 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 --port 22380 raftexample --id 3 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 --port 32380 ``` A fourth node with ID 4 can be added by issuing a POST: ```sh curl -L http://127.0.0.1:12380/4 -XPOST -d http://127.0.0.1:42379 ``` Then the new node can be started as the others were, using the --join option: ```sh raftexample --id 4 --cluster http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379,http://127.0.0.1:42379 --port 42380 --join ``` The new node should join the cluster and be able to service key/value requests. We can remove a node using a DELETE request: ```sh curl -L http://127.0.0.1:12380/3 -XDELETE ``` Node 3 should shut itself down once the cluster has processed this request. ## Design The raftexample consists of three components: a raft-backed key-value store, a REST API server, and a raft consensus server based on etcd's raft implementation. The raft-backed key-value store is a key-value map that holds all committed key-values. The store bridges communication between the raft server and the REST server. Key-value updates are issued through the store to the raft server. The store updates its map once raft reports the updates are committed. The REST server exposes the current raft consensus by accessing the raft-backed key-value store. A GET command looks up a key in the store and returns the value, if any. A key-value PUT command issues an update proposal to the store. The raft server participates in consensus with its cluster peers. When the REST server submits a proposal, the raft server transmits the proposal to its peers. When raft reaches a consensus, the server publishes all committed updates over a commit channel. For raftexample, this commit channel is consumed by the key-value store. --- ### Overview (`third_party/coreos-etcd/hack/benchmark/README.md`) ## Usage Benchmark 3-member etcd cluster to get its read and write performance. ## Instructions 1. Start 3-member etcd cluster on 3 machines 2. Update `$leader` and `$servers` in the script 3. Run the script in a separate machine ## Caveat 1. Set environment variable `GOMAXPROCS` as the number of available cores to maximize CPU resources for both etcd member and bench process. 2. Set the number of open files per process as 10000 for amounts of client connections for both etcd member and benchmark process. --- ### Overview (`third_party/coreos-etcd/hack/insta-discovery/README.md`) Starts a cluster via the discovery service locally. Useful for testing. --- ### Overview (`third_party/coreos-etcd/hack/kubernetes-deploy/README.md`) # etcd on Kubernetes This is an example setting up etcd as a set of pods and services running on top of kubernetes. Using: ``` $ kubectl create -f etcd.yml services/etcd-client pods/etcd0 services/etcd0 pods/etcd1 services/etcd1 pods/etcd2 services/etcd2 $ # now deploy a service that consumes etcd, such as vulcand $ kubectl create -f vulcand.yml ``` TODO: - create a replication controller like service that knows how to add and remove nodes from the cluster correctly - use kubernetes secrets API to configure TLS for etcd clients and peers --- ### Overview (`third_party/coreos-etcd/hack/patch/README.md`) # ./hack/patch/cherrypick.sh Handles cherry-picks of PR(s) from etcd master to a stable etcd release branch automatically. ## Setup Set the `UPSTREAM_REMOTE` and `FORK_REMOTE` environment variables. `UPSTREAM_REMOTE` should be set to git remote name of `github.com/coreos/etcd`, and `FORK_REMOTE` should be set to the git remote name of the forked etcd repo (`github.com/${github-username}/etcd`). Use `git remotes -v` to look up the git remote names. If etcd has not been forked, create one on github.com and register it locally with `git remote add ...`. ``` export UPSTREAM_REMOTE=origin export FORK_REMOTE=${github-username} export GITHUB_USER=${github-username} ``` Next, install hub from https://github.com/github/hub ## Usage To cherry pick PR 12345 onto release-3.2 and propose is as a PR, run: ```sh ./hack/patch/cherrypick.sh ${UPSTREAM_REMOTE}/release-3.2 12345 ``` To cherry pick 12345 then 56789 and propose them togther as a single PR, run: ``` ./hack/patch/cherrypick.sh ${UPSTREAM_REMOTE}/release-3.2 12345 56789 ``` --- ### Overview (`third_party/coreos-etcd/hack/tls-setup/README.md`) This demonstrates using Cloudflare's [cfssl](https://github.com/cloudflare/cfssl) to easily generate certificates for an etcd cluster. Defaults generate an ECDSA-384 root and leaf certificates for `localhost`. etcd nodes will use the same certificates for both sides of mutual authentication, but won't require client certs for non-peer clients. **Instructions** 1. Install git, go, and make 2. Amend https://github.com/coreos/etcd/blob/master/hack/tls-setup/config/req-csr.json - IP's currently in the config should be replaced/added with IP addresses of each cluster node, please note 127.0.0.1 is always required for loopback purposes: ```json Example: { "CN": "etcd", "hosts": [ "3.8.121.201", "46.4.19.20", "127.0.0.1" ], "key": { "algo": "ecdsa", "size": 384 }, "names": [ { "O": "autogenerated", "OU": "etcd cluster", "L": "the internet" } ] } ``` 3. Run `make` to generate the certs --- ### Overview (`third_party/coreos-etcd/contrib/systemd/etcd2-backup-coreos/README.md`) # etcd2-backup-coreos Remote backup and multi-node restore services for etcd2 clusters on CoreOS Linux. **Warning:** This package is only intended for use on CoreOS Linux. ## Terminology **Founding member** : The node which is the first member of the new recovered cluster. It is this node's rclone backup data (only) that will be used to restore the cluster. The rest of the nodes will join the cluster with no data, and simply catch up with the **founding member**. ## Configuration Before installing etcd2-backup, configure `30-etcd2-backup-restore.conf`: ``` [Service] Environment="ETCD_RESTORE_MASTER_ADV_PEER_URLS=" Environment="RCLONE_ENDPOINT=remote-name:path/to/backups" ``` Assuming a deployment to CoreOS with etcd2, only change: * `ETCD_RESTORE_MASTER_ADV_PEER_URLS` This is the new advertised peer url of the new etcd2 node that will be the founding member of the new restored cluster. We will call this node the **founding member**. * `RCLONE_ENDPOINT` The rclone endpoint to which backups will be stored. Feel free to point any number of machines at the same RCLONE_ENDPOINT, path and all. Backups for each machine are stored in a sub-folder named with the machine ID (%m in systemd parlance) * `./rclone.conf` The rclone configuration file which will be installed. Must list a `[section]` which matches `RCLONE_ENDPOINT`'s remote-name component. An easy way to generate this config file is to [install rclone](http://rclone.org/install/) on a local machine. Then follow the [configuration instructions](http://rclone.org/docs/) to generate an `rclone.conf` file. To adjust backup frequency, edit `./etcd2-backup.timer` ## Installation Once those things are configured, run `./build`. The `build` script generates a tarball for copying to CoreOS instances. The tarball contains the `etcd2-backup-install` script. After extracting the contents of the tar file and running the install script, three new systemd services are added. One service, `etcd2-backup`, performs periodic etcd backups, while the other two services, `etcd2-restore` and `etcd2-join`, handle restore procedures. * `etcd2-backup.service` A oneshot service which calls `etcdctl backup` and syncs the backups to the rclone endpoint (using an rclone container, of course). `etcd2-backup.timer` is responsible for periodically running this service. * `etcd2-restore.service` A oneshot service which wipes all etcd2 data and restores a single-node cluster from the rclone backup. This is for restoring the **founding member** only. * `etcd2-join.service` A oneshot service which wipes all etcd2 data and re-joins the new cluster. This is for adding members **after** the **founding member** has succesfully established the new cluster via `etcd2-restore.service` ## Recovery This assumes that the cluster has lost quorum and is not recoverable. Otherwise try to heal the cluster first. ### Backup Freshness Two factors contribute to the relative freshness or staleness of a backup. The `etcd2-backup.timer` takes a backup every 30 seconds by default, and the etcd `snapshot-count` option controls how many transactions are committed between each write of the snapshot to permanent storage. Given those parameters, we can compute the upper bound on the outdatedness of a backup. Assumptions: * transaction rate is a constant `1000 transactions / second` * `etcd2-backup.timer` is configured for a 30 second interval * `etcd2 snapshot-count=10000` ``` max-missed-seconds= (10000 transactions / (1000 transactions / second)) + 30 seconds = 40 seconds ``` ### Recovery Procedure 1. Make sure `etcd2.service` and `etcd2-backup.timer` are stopped on all nodes in the cluster 2. Restore the **founding member** by starting `etcd2-restore.service` and then, if successful, `etcd2.service` 3. Restore the rest of the cluster **one at a time**. Start `etcd2-join.service`, and then, if successful, `etcd2.service`. Please verify with `etcdctl cluster-health` that the expected set of nodes is present and healthy after each node joins. 4. Verify that the data is sane (enough). If so, kick off `etcd2-backup.timer` on all nodes and, hopefully, go back to bed. ## Retroactively change the founding member It is necessary to change the cluster's founding member in order to restore a cluster from any other node's data. Change the value of `ETCD_RESTORE_MASTER_ADV_PEER_URLS` in `30-etcd2-backup-restore.conf` to the advertised peer url of the new founding member. Repeat the install process above on all nodes in the cluster, then proceed with the [recovery procedure](README.md#recovery-procedure). ## Example Let's pretend that we have an initial 3 node CoreOS cluster that we want to back up to S3. | ETCD_NAME | ETCD_ADVERTISED_PEER_URL | | ------------- |:-------------:| | e1 | http://172.17.4.51:2379 | | e2 | http://172.17.4.52:2379 | | e3 | http://172.17.4.53:2379 | In the event that the cluster fails, we want to restore from `e1`'s backup ## Configuration ``` [Service] Environment="ETCD_RESTORE_MASTER_ADV_PEER_URLS=http://172.17.4.51:2379" Environment="RCLONE_ENDPOINT=s3-testing-conf:s3://etcd2-backup-bucket/backups" ``` The `./rclone.conf` file must contain a `[section]` matching `RCLONE_ENDPOINTS`'s remote-name component. ``` [s3-testing-conf] type = s3 access_key_id = xxxxxxxx secret_access_key = xxxxxx region = us-west-1 endpoint = location_constraint = ``` ## Installation ```sh cd etcd2-backup ./build scp etcd2-backup.tgz core@e1:~/ ssh core@e1 e1 $ mkdir -p ~/etcd2-backup e1 $ mv etcd2-backup.tgz etcd2-backup/ e1 $ cd etcd2-backup e1 $ tar zxvf ~/etcd2-backup.tgz e1 $ ./etcd2-backup-install # Only do the following two commands if this node should generate backups e1 $ sudo systemctl enable etcd2-backup.timer e1 $ sudo systemctl start etcd2-backup.timer e1 $ exit ``` Now `e1`'s etcd data will be backed up to `s3://etcd2-backup-bucket/backups//` according to the schedule described in `etcd2-backup.timer`. Repeat the process for `e2` and `e3`. To stop a node from generating backups, omit enabling and starting `etcd2-backup.timer`. ## Restore the cluster Let's assume that a mischievous friend decided it would be a good idea to corrupt the etcd2 data-dir on ALL of the nodes (`e1`,`e2`,`e3`). Simply restore the cluster from `e1`'s backup. Here's how to recover: ```sh # First, ENSURE etcd2 and etcd2-backup are not running on any nodes for node in e{1..3};do ssh core@$node "sudo systemctl stop etcd2.service etcd2-backup.{timer,service}" done ssh core@e1 "sudo systemctl start etcd2-restore.service && sudo systemctl start etcd2.service" for node in e{2..3};do ssh core@$node "sudo systemctl start etcd2-join.service && sudo systemctl start etcd2.service" sleep 10 done ``` After e2 and e3 finish catching up, the cluster should be back to normal. ## Migrate the cluster The same friend who corrupted the etcd2 data-dirs decided that to have more fun. This time, the friend dumps coffee on the machines hosting `e1`, `e2` and `e3`. There is a horrible smell, and the machines are dead. Luckily, there's a new 3-node etcd2 cluster ready to go, along with the S3 backup for `e1` from the old cluster. The new cluster configuration looks like this. Assume that etcd2-backup is not installed. (If it is, make sure it's not running on any nodes) | ETCD_NAME | ETCD_ADVERTISED_PEER_URL | | ------------- |:-------------:| | q1 | http://172.17.8.201:2379 | | q2 | http://172.17.8.202:2379 | | q3 | http://172.17.8.203:2379 | We will assume `q1` is the chosen founding member, though picking any node is fine. ## Migrate the remote backup First, copy the backup from `e1`'s backup folder to `q1`'s backup folder. I will show the S3 example. ```sh # Make sure to remove q1's backup directory, if it exists already aws s3 rm --recursive s3://etcd2-backup-bucket/backups/ aws s3 cp --recursive s3://etcd2-backup-bucket/backups/ s3://etcd2-backup-bucket/backups/ ``` ## Configure the New Cluster ``` [Service] Environment="ETCD_RESTORE_MASTER_ADV_PEER_URLS=http://172.17.8.201:2379" Environment="RCLONE_ENDPOINT=s3-testing-conf:s3://etcd2-backup-bucket/backups" ``` Since this is a new cluster, each new node will have a new `machine-id` and will not clobber the backups from the old cluster, even though `RCLONE_ENDPOINT` is the same for both the old `e` cluster and the new `q` cluster. ## Installation We first want to install the configured etcd2-backup package on all nodes, but not start any services yet. ```sh cd etcd2-backup ./build for node in q{1..3};do scp etcd2-backup.tgz core@$node:~/ ssh core@$node "mkdir -p ~/etcd2-backup" ssh core@$node "mv etcd2-backup.tgz etcd2-backup/" ssh core@$node " cd etcd2-backup" ssh core@$node " tar zxvf ~/etcd2-backup.tgz" ssh core@$node " ./etcd2-backup-install" done ``` ## Migrate the Cluster With `q1` as the founding member. ```sh # First, make SURE etcd2 and etcd2-backup are not running on any nodes for node in q{1..3};do ssh core@$node "sudo systemctl stop etcd2.service" done ssh core@q1 "sudo systemctl start etcd2-restore.service && sudo systemctl start etcd2.service" for node in q{2..3};do ssh core@$node "sudo systemctl start etcd2-join.service && sudo systemctl start etcd2.service" sleep 10 done ``` After confirming the cluster has migrated properly, start and enable `etcd2-backup.timer` on at least one node. ```sh ssh core@q1 "sudo systemctl enable etcd2-backup.service && sudo systemctl start etcd2-backup.service" ``` There should now be periodic backups going to: `s3://etcd2-backup-bucket/backups/` ## Words of caution 1. Notice the `sleep 10` commands that follow starting `etcd2-join.service` and then `etcd2.service`. This sleep is there to allow the member that joined to cluster time to catch up on the cluster state before we attempt to add the next member. This involves sending the entire snapshot over the network. If the dataset is large, the network between nodes is slow, disks are already bogged down, or the system is otherwise overutilized, try increasing the sleep. In the case of large data sets, it is recommended to copy the data directory produced by `etcd2-restore` on the founding member to the other nodes before running `etcd2-join` on them. This will avoid etcd transferring the entire snapshot to every node after it joins the cluster. 2. It is not recommended clients be allowed to access the etcd2 cluster **until** all members have been added and finished catching up. --- ### Overview (`third_party/coreos-etcd/contrib/systemd/etcd3-multinode/README.md`) # etcd3 multi-node cluster Here's how to deploy etcd cluster with systemd. ## Set up data directory etcd needs data directory on host machine. Configure the data directory accessible to systemd as: ``` sudo mkdir -p /var/lib/etcd sudo chown -R root:$(whoami) /var/lib/etcd sudo chmod -R a+rw /var/lib/etcd ``` ## Write systemd service file In each machine, write etcd systemd service files: ``` cat > /tmp/my-etcd-1.service < /tmp/my-etcd-2.service < /tmp/my-etcd-3.service < rainbond-console /console/teams/{team}/components/{component}/k8s-attributes -> rainbond /v2/tenants/{tenant}/services/{service}/k8s-attributes -> worker 转换为 Kubernetes Container.Command / Container.Args ``` ### 1.2 现有基础 - `rainbond` 已有 `component_k8s_attributes` 通用表，字段包含 `name`、`save_type`、`attribute_value`。 - `rainbond` 已定义 `cmd` 和 `args` 属性名，并在 `worker/appm/conversion/version.go` 中应用到容器 `Command` 和 `Args`。 - `rainbond-console` 通过 `console.services.k8s_attribute` 保存 console 侧记录并同步到 region。 - `rainbond-ui` 的组件 Kubernetes 属性页已支持 JSON、YAML、string、boolean 等编辑模式。 ### 1.3 核心需求 当前 `cmd` 被 UI 当作字符串输入，region worker 在解析失败时使用 `strings.Split(value, " ")` 兜底，导致包含空格或 shell 结构的一行命令被错误拆分，例如： ```text /entrypoint.sh;while true;do echo hello >/dev/null;sleep 1;done ``` 目标是支持 `cmd` 和 `args` 以 YAML 数组格式录入、存储和解析，避免任何空格切割。 ## 二、用户旅程（MUST — 禁止跳过） ### 2.1 用户操作流程 - 用户如何配置/触发该功能？（必须回答，若需要 UI 则列出页面） 用户进入组件详情 -> Kubernetes 属性视图，点击添加属性，选择 `cmd` 或 `args`，在 YAML 编辑器中录入数组。 - 用户如何查看状态/结果？ Kubernetes 属性列表中 `cmd` 和 `args` 按 YAML 类型展示，用户点击编辑可看到原 YAML 内容。 - 管理员/审批人如何操作？ 不涉及额外管理员或审批流程，沿用现有组件配置权限。 ### 2.2 页面原型 - 页面：组件详情 -> Kubernetes 属性视图。 - 弹窗/抽屉：添加/编辑 Kubernetes 属性抽屉。 - 表单交互： - `cmd` 示例： ```yaml - /entrypoint.sh ``` - `args` 示例： ```yaml - /entrypoint.sh;while true;do echo hello >/dev/null;sleep 1;done ``` - 提示文案说明每一行 `- ...` 是 Kubernetes argv 的一个元素，不会按空格切割。 ### 2.3 外部系统交互 不涉及 webhook、回调、通知或第三方集成。最终变更体现在 Kubernetes Pod template 的 `containers[].command` 和 `containers[].args` 字段。 ## 三、整体架构设计 ### 3.1 系统架构图 ```text UI YAML editor saves { name: "cmd"|"args", save_type: "yaml", attribute_value: "- ..." } | Console k8s_attribute service stores YAML string and forwards unchanged | Rainbond region API stores save_type/attribute_value | Worker conversion parses YAML/JSON sequence into []string | Kubernetes Container.Command / Container.Args ``` ### 3.2 核心流程 1. UI 将 `cmd` 和 `args` 纳入 YAML 字段列表，并从 string 字段中移除 `cmd`。 2. UI 添加 `args` 到可选 Kubernetes 属性列表，并更新 `cmd/args` 提示示例。 3. console 对 `save_type=yaml` 保持原文存储和透传，不做空格拆分。 4. region worker 对 `cmd/args` 只解析 YAML/JSON sequence；旧 `save_type=string` 或无法解析的历史值整体作为单个 argv 元素。 5. compose 检测等内部来源继续可用，但优先保存为 YAML sequence，保证与 UI 契约一致。 ## 四、数据模型设计 ### 4.1 新增数据库表 不新增数据库表。 ### 4.2 数据关系 继续使用 `component_k8s_attributes`： ```text tenant_id component_id name cmd / args save_type yaml attribute_value YAML sequence string ``` 旧数据兼容规则： - JSON 数组旧值继续按数组解析。 - YAML 数组新值按数组解析。 - 无法解析为数组的历史字符串不再空格切割，整体作为一个参数。 ## 五、API设计 ### 5.1 接口列表 沿用现有接口，不新增路由： - `GET /console/teams/{team_name}/components/{service_alias}/k8s-attributes` - `POST /console/teams/{team_name}/components/{service_alias}/k8s-attributes` - `PUT /console/teams/{team_name}/components/{service_alias}/k8s-attributes/{name}` - region 对应 `/v2/tenants/{tenant_name}/services/{service_alias}/k8s-attributes` ### 5.2 请求/响应结构 新增推荐请求格式： ```json { "attribute": { "name": "args", "save_type": "yaml", "attribute_value": "- /entrypoint.sh;while true;do echo hello >/dev/null;sleep 1;done\n" } } ``` `cmd` 同样使用 `save_type=yaml`，值必须表达字符串数组。 ## 六、核心实现设计 ### 6.1 关键逻辑 - `rainbond`： - 新增或提取 `parseStringSequenceAttribute` 类辅助逻辑，按 YAML/JSON 数组解析 `cmd/args`。 - 当解析结果不是字符串数组时返回错误，避免静默生成错误 Pod spec。 - 对旧字符串值整体保留为 `[]string{value}`，不再使用 `strings.Split(value, " ")`。 - `rainbond-console`： - `app_check_service` 中自动保存 compose `command/args` 时改为 YAML sequence，`save_type=yaml`。 - `k8s_attribute_service` 对用户提交的 `cmd/args` 做轻量校验：必须是 YAML/JSON 数组，且每项是字符串。 - `rainbond-ui`： - `DEFAULT_ATTRIBUTE_FIELDS` 增加 `args`。 - `YAML_FIELDS` 增加 `cmd`、`args`；`STRING_FIELDS` 移除 `cmd`。 - 更新 `TooltipValueArr` 和输入提示文案，突出“不按空格切割”。 ### 6.2 复用现有代码 - 复用现有 `CodeMirrorForm` YAML 编辑器。 - 复用现有 `save_type=yaml` 存储和展示逻辑。 - 复用现有 console -> region API 透传链路。 ## 七、实施计划 ### 跨层覆盖检查（MUST） - [ ] Go (rainbond): 需要 — worker 解析 `cmd/args` YAML/JSON sequence，禁止空格拆分 fallback，补 Go 测试。 - [ ] Python (console): 需要 — 校验并保存 `cmd/args` YAML sequence，内部检测来源也改为 YAML 存储，补 Python 测试。 - [ ] React (rainbond-ui): 需要 — 组件 Kubernetes 属性页增加 `args`，把 `cmd/args` 切为 YAML 编辑并更新提示。 - [ ] Plugin frontend (enterprise-base): 不涉及 — 该功能在主平台组件视图。 - [ ] Plugin backend (plugin-template): 不涉及 — 无插件 API 变更。 ### Sprint 1: Region 解析契约 #### Task 1.1: 补充 cmd/args 解析测试 - 仓库：rainbond - 文件：`worker/appm/conversion/version_test.go` 或新增聚焦测试文件 - 实现内容：覆盖 YAML 数组、JSON 数组、单个 shell 字符串 fallback。 - 验收标准：旧空格切割场景失败后修复通过。 #### Task 1.2: 实现 YAML/JSON sequence 解析 - 仓库：rainbond - 文件：`worker/appm/conversion/version.go` - 实现内容：替换 `strings.Split` fallback，解析失败时整体作为单元素。 - 验收标准：`go test ./worker/appm/conversion -run CmdArgs -v` 通过。 ### Sprint 2: Console 保存与校验 #### Task 2.1: 补充 k8s attribute service 测试 - 仓库：rainbond-console - 文件：`console/tests/k8s_attribute_service_test.py` - 实现内容：验证 `cmd/args` YAML 数组保存透传，非数组报错。 - 验收标准：相关 pytest 通过。 #### Task 2.2: 实现 console 校验和 compose 保存格式 - 仓库：rainbond-console - 文件：`console/services/k8s_attribute.py`、`console/services/app_check_service.py` - 实现内容：`cmd/args` 入库前规范为 YAML sequence，内部检测保存使用 `save_type=yaml`。 - 验收标准：相关 pytest 通过。 ### Sprint 3: UI 编辑体验 #### Task 3.1: 修改 Kubernetes 属性页面 - 仓库：rainbond-ui - 文件：`src/pages/Component/kubernets.js` - 实现内容：`cmd/args` 使用 YAML 编辑器，新增示例和提示。 - 验收标准：`yarn build` 通过。 #### Task 3.2: 更新 i18n 文案 - 仓库：rainbond-ui - 文件：`src/locales/zh-CN/*.js`、`src/locales/en-US/*.js` - 实现内容：更新 `cmd/args` 提示文案。 - 验收标准：页面引用的文案 key 完整。 ## 八、关键参考代码 | 功能 | 文件 | 说明 | |------|------|------| | region API 路由 | `api/api_routers/version2/v2Routers.go:610` | `/k8s-attributes` 路由 | | region 属性模型 | `api/model/component.go:276` | `ComponentK8sAttribute` | | worker 容器转换 | `worker/appm/conversion/version.go:480` | `cmd/args` 应用到 Kubernetes Container | | console 服务 | `console/services/k8s_attribute.py:15` | console 保存并同步 region | | console 视图 | `console/views/k8s_attribute.py:14` | console k8s attribute API | | UI 属性页面 | `src/pages/Component/kubernets.js:25` | Kubernetes 属性可选字段和编辑器选择 | --- ### Third Party Coreos Etcd Etcdctl Doc Mirror Maker (`third_party/coreos-etcd/etcdctl/doc/mirror_maker.md`) ## Mirror Maker Mirror maker mirrors a prefix in the key-value space of an etcd cluster into another prefix in another cluster. Mirroring is designed for copying configuration to various clusters distributed around the world. Mirroring usually has very low latency once it completes synchronizing with the initial state. Mirror maker utilizes the etcd watcher facility to immediately inform the mirror of any key modifications. Based on our experiments, the network latency between the mirror maker and the two clusters accounts for most of the latency. If the network is healthy, copying configuration held in etcd to the mirror should take under one second even for a world-wide deployment. If the mirror maker fails to connect to one of the clusters, the mirroring will pause. Mirroring can be resumed automatically once connectivity is reestablished. The mirroring mechanism is unidirectional. Data under the destination cluster’s mirroring prefix should be treated as read only. The mirror maker only mirrors key-value pairs; metadata, such as version number or modification revision, is discarded. However, mirror maker still attempts to preserve update ordering during normal operation, but there is no ordering guarantee during initial sync nor during failure recovery following network interruption. As a rule of thumb, the ordering of the updates on the mirror should not be considered reliable. ``` +-------------+ | | | source | +-----------+ | cluster +----> | mirror | | | | maker | +-------------+ +---+-------+ | v +-------------+ | | | mirror | | cluster | | | +-------------+ ``` Mirror-maker is a built-in feature of [etcdctl][etcdctl]. [etcdctl]: ../README.md --- --- METRICS --- - Files Extracted: 28 - Estimated Token Budget: ~34141 tokens - Canonical Reference: https://codewiki.google/github.com/goodrain/rainbond