## 1. Project Overview & Quickstart (etcd-io/OWNERS) # OWNERS Open-source repository etcd-io/OWNERS ### Repository Details - **Repository:** [etcd-io/OWNERS](https://github.com/etcd-io/OWNERS) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (etcd-io/website) ## File: README.md # etcd.io The [etcd.io][] website, built using [Hugo][] and hosted on [Netlify][]. ## Cloud build Visit [https://gitpod.io/#https://github.com/etcd-io/website](https://gitpod.io/#https://github.com/etcd-io/website) to launch a [Gitpod.io](https://gitpod.io) IDE that will allow you to build, preview and make changes to this repo. ## Local build To build and serve the site, you'll need the latest [LTS release][] of **Node**. Like Netlify, we use **[nvm][]**, the Node Version Manager, to install and manage Node versions: ```console $ nvm install --lts ``` ### Setup 1. Clone this repo. 2. From a terminal window, change to the cloned repo directory. 3. Get NPM packages and git submodules, including the the [Docsy][] theme: ```console $ npm install ``` ### Build or serve the site To locally serve the site at [localhost:8888][], run the following command: ```console $ npm run serve ``` > **NOTE:** If you want to **check links** locally, you (temporarily) might have > some extra setup to do. For details, see issue > [#459](https://github.com/etcd-io/website/issues/459). To build and check links, run these commands: ```console $ npm run build $ npm run check-links ``` ## Contributing We welcome issues and PRs! For details, see [Contributing][]. If you submit a PR, Netlify will automatically create a [deploy preview][] so that you can view your changes. Once merged, Netlify automatically deploys to the production site [etcd.io][]. ## Releasing a new version of the etcd documentation > **IMPORTANT**: this section is out-of-date, see issue [#406][]. Follow these steps to add documentation for a newly released version of etcd, vX.Y: * Recursively copy [content/docs/next](content/docs/next) into `content/docs/vX.Y`, where `vX.Y` is the newly released version of etcd. For example: ```bash cp -r content/en/docs/next content/en/docs/v3.6 ``` * In the `_index.md` file at the root of the new directory, update the frontmatter to reflect the new version: ``` --- title: etcd version X.Y weight: 1000 cascade: version: vX.Y --- ``` * Add the version to the `params.versions.all` array in the [config.toml](config.toml) configuration file. * If the version is meant to be the latest version of etcd, change the `params.latest_stable_version` parameter to the desired new version. * Submit a pull request with the changes. [#406]: https://github.com/etcd-io/website/issues/406 [Contributing]: https://etcd.io/community/#contributing [deploy preview]: https://www.netlify.com/blog/2016/07/20/introducing-deploy-previews-in-netlify/ [Docsy]: https://www.docsy.dev [etcd.io]: https://etcd.io [Hugo]: https://gohugo.io [localhost:8888]: http://localhost:8888 [LTS release]: https://nodejs.org/en/download/ [Netlify]: https://netlify.com [nvm]: https://github.com/nvm-sh/nvm/blob/master/README.md#installing-and-updating --- ## File: content/en/blog/2014/etcd-freeBSD.md --- title: FreeBSD date: 2014-01-23 --- --- _This is an adaptation of a page previously found in the Platforms section of the documentation which described etcd deployments on various platform services. The original page was authored by [**Bartłomiej Rutkowski** and others](https://github.com/etcd-io/etcd/commits/6c08437ec330f84f78a59f7772884c7ef5374247/Documentation/platforms/freebsd.md)._ --- Starting with version 0.1.2 both etcd and etcdctl have been ported to FreeBSD and can be installed either via packages or ports system. Their versions have been recently updated to 0.2.0 so now etcd and etcdctl can be enjoyed on FreeBSD 10.0 (RC4 as of now) and 9.x, where they have been tested. They might also work when installed from ports on earlier versions of FreeBSD, but it is untested; caveat emptor. ## Installation ### Using pkgng package system 1. If pkg­ng is not installed, install it with command `pkg` and answering 'Y' when asked. 2. Update the repository data with `pkg update`. 3. Install etcd with `pkg install coreos-etcd coreos-etcdctl`. 4. Verify successful installation by confirming `pkg info | grep etcd` matches: ``` r@fbsd­10:/ # pkg info | grep etcd coreos­etcd­0.2.0 Highly­available key value store and service discovery coreos­etcdctl­0.2.0 Simple commandline client for etcd r@fbsd­10:/ # ``` 5. etcd and etcdctl are ready to use! For more information about using pkgng, please see: http://www.freebsd.org/doc/handbook/pkgng­intro.html ### Using ports system 1. If ports is not installed, install with `portsnap fetch extract` (it may take some time depending on hardware and network connection). 2. Build etcd with `cd /usr/ports/devel/etcd && make install clean`. There will be an option to build and install documentation and etcdctl with it. 3. If etcd wasn't installed with etcdctl, it can be built later with `cd /usr/ports/devel/etcdctl && make install clean`. 4. Verify successful installation by confirming `pkg info | grep etcd` matches: ``` r@fbsd­10:/ # pkg info | grep etcd coreos­etcd­0.2.0 Highly­available key value store and service discovery coreos­etcdctl­0.2.0 Simple commandline client for etcd r@fbsd­10:/ # ``` 5. etcd and etcdctl are ready to use! For more information about using ports system, please see: https://www.freebsd.org/doc/handbook/ports­using.html ## Issues If there are any issues with the build/install procedure or there's a problem that is local to FreeBSD only (for example, by not being able to reproduce it on any other platform, like OSX or Linux), please send a problem report using this page for more information: http://www.freebsd.org/send­pr.html --- ## File: content/en/blog/2017/etcd-container-linux-with-systemd.md --- title: Container Linux with systemd date: 2017-05-23 --- --- _This is an adaptation of a page previously found in the Platforms section of the documentation which described etcd deployments on various platform services. The original page was authored by [**Gyuho Lee**](https://github.com/etcd-io/etcd/commits/8b290c680a5d4b4067c4d91b33f70c5506f654f0/Documentation/platforms/container-linux-systemd.md)._ --- The following blog post shows how to run etcd with [systemd][systemd-docs] under [Container Linux][container-linux-docs]. ## Provisioning an etcd cluster Cluster bootstrapping in Container Linux is simplest with [Ignition][container-linux-ignition]; `coreos-metadata.service` dynamically fetches the machine's IP for discovery. Note that etcd's discovery service protocol is only meant for bootstrapping, and cannot be used with runtime reconfiguration or cluster monitoring. The [Container Linux Config Transpiler][container-linux-ct] compiles etcd configuration files into Ignition configuration files: ```yaml container-linux-config:norender etcd: version: 3.2.0 name: s1 data_dir: /var/lib/etcd advertise_client_urls: http://{PUBLIC_IPV4}:2379 initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380 listen_client_urls: http://0.0.0.0:2379 listen_peer_urls: http://{PRIVATE_IPV4}:2380 discovery: https://discovery.etcd.io/ ``` `ct` would produce the following Ignition Config: ``` $ ct --platform=gce --in-file /tmp/ct-etcd.cnf {"ignition":{"version":"2.0.0","config"... ``` ```json ignition-config { "ignition":{"version":"2.0.0","config":{}}, "storage":{}, "systemd":{ "units":[{ "name":"etcd-member.service", "enable":true, "dropins":[{ "name":"20-clct-etcd-member.conf", "contents":"[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=\"ETCD_IMAGE_TAG=v3.1.8\"\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n --name=\"s1\" \\\n --data-dir=\"/var/lib/etcd\" \\\n --listen-peer-urls=\"http://${COREOS_GCE_IP_LOCAL_0}:2380\" \\\n --listen-client-urls=\"http://0.0.0.0:2379\" \\\n --initial-advertise-peer-urls=\"http://${COREOS_GCE_IP_LOCAL_0}:2380\" \\\n --advertise-client-urls=\"http://${COREOS_GCE_IP_EXTERNAL_0}:2379\" \\\n --discovery=\"https://discovery.etcd.io/\u003ctoken\u003e\""}]}]}, "networkd":{}, "passwd":{}} ``` To avoid accidental misconfiguration, the transpiler helpfully verifies etcd configurations when generating Ignition files: ```yaml container-linux-config:norender etcd: version: 3.2.0 name: s1 data_dir_x: /var/lib/etcd advertise_client_urls: http://{PUBLIC_IPV4}:2379 initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380 listen_client_urls: http://0.0.0.0:2379 listen_peer_urls: http://{PRIVATE_IPV4}:2380 discovery: https://discovery.etcd.io/ ``` ``` $ ct --platform=gce --in-file /tmp/ct-etcd.cnf warning at line 3, column 2 Config has unrecognized key: data_dir_x ``` See [Container Linux Provisioning][container-linux-provision] for more details. ## etcd 3.x service [Container Linux][container-linux-docs] does not include etcd 3.x binaries by default. Different versions of etcd 3.x can be fetched via `etcd-member.service`. Confirm unit file exists: ``` systemctl cat etcd-member.service ``` Check if the etcd service is running: ``` systemctl status etcd-member.service ``` Example systemd drop-in unit to override the default service settings: ```bash cat > /tmp/20-cl-etcd-member.conf < etcd 3.4.3 lived up to its claims for key-value operations: we observed nothing but strict-serializable consistency for reads, writes, and even multi-key transactions, during process pauses, crashes, clock skew, network partitions, and membership changes. Strict serializable behavior was the default for key-value operations. > Watches appear correct, at least over single keys. So long as compaction does not destroy historical data while a watch isn't running, watches appear to deliver every update to a key in order. Since Jepsen never "passes" software, but rather reports a lack of prohibitive issues, this is a great result from them. In general, it's much better than [other distributed databases](https://jepsen.io/analyses) tested to date. We believe two factors contribute to this positive result to the rigorous Jepsen analysis: ### 1. A simple core A simple solution to a problem space usually results in a robust system. As an example, etcd's Raft implementation is one of the simplest implementations in open source. It focuses on the core state machine of Raft and avoids wall-time and I/O handling in the core library. And, in its API, etcd supports multi-key transactions, but adopts a simple transactional model to keep the system easier to comprehend. ### 2. Rigorous testing practice Early in the development cycle of etcd, we set a goal to [achieve 80% unit test line coverage](https://github.com/etcd-io/etcd/issues/1467) and 90% unit test line coverage over core components like Raft and the MVCC storage engine. As the project progressed, we put effort into ensuring logical path coverage for core components by [introducing fail points](https://github.com/etcd-io/gofail). To embrace the randomness of timing, OS scheduling and asynchronous network I/Os, we also developed continuous integration tests, which run 24x7, to try and produce issues on a running system in a way that unit tests simply cannot. These tests have increased the quality of etcd significantly. ## Identified Issues Of course, no system undergoes Jepsen testing without them reporting some areas where it could be better. etcd had a few: ### Lock Issues One lock implementation issue was found during the test where etcd failed to check the ownership of the lock before the pending lock API call returned. In etcd, a lock acquirer is associated with a session; the acquirer holds the lock until the session expires. When first attempting to acquire the lock, it may be held by someone else. In that case, the etcd server puts the acquirer into a queue, where it must wait until other lock holders release the lock. The problem is that the acquirer's session might expire during this wait time. The consequence of this premature expiration is that the etcd server must check the existence of the session again before returning to the API call. ### Documentation Issues The Jepsen analysis not only covers software correctness but also verifies the documentation claims. The Jepsen author found documentation issues and misleading use case examples of etcd. The previous etcd documentation on consistency and isolation is not clear and might cause confusion. Our documentation describes the consistency model based on [the Wikipedia definition](https://en.wikipedia.org/wiki/Consistency_model), which is also the classification some etcd engineers [learned in University](http://www.cs.cmu.edu/~srini/15-446/S09/lectures/10-consistency.pdf). It also separates the isolation level from consistency level, as there is no general agreement on [how the two should be defined together](https://fauna.com/blog/demystifying-database-systems-part-4-isolation-levels-vs-consistency-levels). The Jepsen author suggests etcd documentation adopt the term "strict serializability" originally defined in the [Linearizability paper published by Herlihy and Wing](https://web.archive.org/web/20201203054309/https://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf). After reading the documentation on other open source projects, as well as listening to feedback from our users, the etcd team agrees with the Jepsen author that the [suggested model and term](https://jepsen.io/consistency) are clearer and easier to understand. Next, the lock API use case example is misleading and might cause runtime issues. Like any other distributed lock system, an etcd lock provides different guarantees than say a local thread level lock. Specifically, an etcd lock only safely guarantees mutual exclusion inside etcd's own keyspace and transactions with revision checking. It provides weaker guarantee when accessing external resources with dependency on timing. Deadlock prevention and lock invalidation is difficult with distributed locks because a distributed lock does not protect resources within the same process, or even the same machine. To address this, distributed locks traditionally rely on a lease and heartbeat mechanism to detect disconnected lock holders and invalidate locks. When a lock holder is disconnected or paused without using revision checking (a fencing token implementation in etcd), it might access the protected resources simultaneously with the new lock holder. More details can be found [in Kleppmann's blog post](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html). The etcd lock documentation fails to point out this difference and provides inaccurate examples of using the lock. We agree with the Jepsen team that the issues must be resolved, and etcd users should be informed with the accurate claims. ## Addressing Identified Issues All the issues mentioned above are being addressed by the etcd team and the Jepsen team. Here is the list of issues and fixes. We appreciate feedback from community over these issues and help the etcd project become better. | Analysis Issues | Github Issue | Status on 1/30 | | -------------------- | ----- | --------- | | Lock implementation | [11408](https://github.com/etcd-io/etcd/pull/11408) | Fixed | Watch API defaulting documentation | [11496](https://github.com/etcd-io/etcd/issues/11496) | Fixed | Consistency documentation | [11474](https://github.com/etcd-io/etcd/pull/11474) | Fixed | Lock documentation | [11490](https://github.com/etcd-io/etcd/pull/11490) | Work in progress ## Future Jepsen analysis is not a one time effort. During the analysis, the Jepsen team set up an extensive testing framework for the etcd project specifically. It also enables etcd team and the community to run those tests anytime and catch problems in the future. We would love to see someone in the etcd community integrate the etcd Jepsen tests directly into the existing etcd release pipeline. We hope to ensure that all future releases of etcd are Jepsen tests approved. Besides Jepsen analysis, etcd community always welcome contributions related to correctness and reliability. We are excited about the results of this testing, and will remain vigilant while building a well engineered and correct product. To learn more read the full [Jepsen report for etcd 3.4.3](https://jepsen.io/analyses/etcd-3.4.3). --- ## File: content/en/blog/2021/announcing-etcd-3.5.md --- title: Announcing etcd 3.5 spelling: cSpell:ignore Gyuho author: "[Gyuho Lee](https://github.com/gyuho), Amazon Web Services" date: 2021-06-15 draft: false --- When we [launched etcd 3.4 back in August 2019][3.4-blog], our focus was on storage backend improvements, non-voting member and pre-vote features. Since then, etcd has become more widely used for various mission critical clustering and database applications and as a result, its feature set has grown more broad and complex. Thus, improving its stability and reliability has been top priority in recent development. Today, we are releasing [etcd 3.5][]. The past two years allowed for extensive iterations in fixing numerous bugs, identifying optimization opportunities at scale, and evolving its surrounding ecosystem. The etcd project also became a [CNCF graduated project](https://www.cncf.io/announcements/2020/11/24/cloud-native-computing-foundation-announces-etcd-graduation/) during this time frame. This release is the result of continuous evolution and grungy, thankless tasks done by the etcd community. In this blog post, we review the most notable changes to etcd 3.5, and present a project road map for future releases. For a complete list of changes, see [CHANGELOG 3.5][]. For more updates, follow us on Twitter [@etcdio][]. To get etcd, see [Install][]. ## Security Given that etcd often handles sensitive data, improving and maintaining security posture is our highest priority. In order to have a comprehensive understanding of etcd security landscape, we completed third-party security audits: The first report was published in February 2020, where we identified (and fixed) various edge cases and high severity issues. For more information, see [security audit report](https://github.com/etcd-io/etcd/blob/master/security/SECURITY_AUDIT.pdf). To adhere to the highest levels of security best practices, etcd now has a [security release process](https://github.com/etcd-io/etcd/blob/master/security/security-release-process.md), and runs automated tests with static analysis tools, such as `errcheck`, `ineffassign`, and others. ## Features The **migration to structured logging is complete**. etcd now defaults to [zap](https://github.com/uber-go/zap) logger that has a reflection-free, zero-allocation JSON encoder. We have deprecated [capnslog](https://github.com/coreos/pkg/issues/57) that logged with reflection-based serialization. **etcd now supports built-in log rotation** that configures rotate thresholds, compression algorithms, etc. For more information, see [hexfusion@ of Red Hat's code change](https://github.com/etcd-io/etcd/pull/12774). **etcd now emits more detailed tracing information for expensive requests**, such as: ``` { "caller":"traceutil/trace.go:116", "msg":"trace[123] range", "detail":"{ range_begin:foo; range_end:fooo; response_count:100000; response_revision:191496;}", "duration":"132.449773ms", "start":"...:32.611-0700", "end":"...:32.744-0700", "steps":[ "trace[123] step 'range keys from bolt db' (duration: 92.521911ms)", "trace[123] step 'filter and sort the key-value pairs' (duration: 22.789099ms)"]} ``` This provides a very useful signal for understanding the lifetime of a request spanning multiple etcd server components. See [YoyinZyc@'s code change (from Google)](https://github.com/etcd-io/etcd/pull/11179). Each etcd cluster maintains its own cluster version, a value agreed on by the quorum of the cluster. Previously, downgrading such cluster versions (e.g., etcd minor version from 3.5 to 3.4) was not supported in order to protect against incompatible changes. Let's say we allow 3.3 node to join 3.4 cluster and send a lease checkpoint request to the leader, which was introduced only in etcd 3.4. When 3.3 node receives the lease checkpoint request, it fails to handle the unknown apply request (see [etcd server apply code](https://github.com/etcd-io/etcd/blob/v3.3.25/etcdserver/apply.go#L119-L170)). However, one might not use such a lease checkpointer feature and be **willing to risk incompatible changes in order to perform emergency rollbacks** (e.g., major defect in new etcd versions). To ensure such rollbacks are easy and reliable, we've added a **downgrade API that validates, enables, and cancels etcd version downgrades**. For more information, see [YoyinZyc@ of Google's code change ](https://github.com/etcd-io/etcd/pull/11715). By requiring quorum agreement, etcd cluster membership applies with the same level of consistency as writes. Previously however, the member list call was directly served from the server's local data, which could be stale. Now, **etcd serves the member list with a linearizable guarantee — if the server is disconnected from the quorum, the member list call will fail**. See [jingyih@ of Google's code change](https://github.com/etcd-io/etcd/pull/11639). **gRPC gateway endpoint is now stable under `/v3/*`**. gRPC gateway generates an HTTP API to make etcd gRPC-based HTTP/2 protocol accessible via HTTP/1, such as: ``` curl -X POST -L http://localhost:2379/v3/kv/put -d '{"key": "Zm9v", "value": "YmFy"}' ``` **etcd client now uses the latest gRPC, v1.32.0, requires a new import path `"go.etcd.io/etcd/client/v3"`**, and migrates balancer implementation upstream. For more information, see [ptabor@ of Google's code change](https://github.com/etcd-io/etcd/pull/12671). ## Bug fixes etcd reliability and correctness are of utmost importance. That is why we backport all critical bug fixes to previous etcd releases. The following are the notable bugs we identified and fixed during etcd 3.5 development: 1. Lease objects piling up caused memory leaks, and the solution was to clear the expired lease queue in the old leader. For details, see [tangcong@ of Tencent's fix](https://github.com/etcd-io/etcd/pull/11731). 1. Ongoing compact operation caused deadlock in mvcc storage layer. For details, see [tangcong@ of Tencent's fix](https://github.com/etcd-io/etcd/pull/11817). 1. etcd server restart had redundant backend database open operations and as a result, reloading 40-million keys took over 5-minutes, and the solution reduced the restart time by half. For details, see [tangcong@ of Tencent's fix](https://github.com/etcd-io/etcd/pull/11779). 1. If etcd crashed before completing defragmentation, the next defragment operation might have read the corrupted file. The solution was to ignore and overwrite the existing file. See [jpbetz@ of Google's fix](https://github.com/etcd-io/etcd/pull/11613). 1. Client cancelling watch did not signal the server to create leaky watchers. The solution was to explicitly send a cancel request to the server. See [jackkleeman@ of Apple's fix](https://github.com/etcd-io/etcd/pull/11850). ## Performance Kubernetes, the most prominent user of etcd, queries the entire keyspace to list and watch its cluster resources. This range query happens whenever the resource is not found in the `kube-apiserver` reflector cache (e.g., requested etcd revision has been compacted, see [kube-apiserver v1.21 code](https://github.com/kubernetes/kubernetes/blob/v1.21.0/staging/src/k8s.io/client-go/tools/cache/reflector.go#L302-L312)), often causing slowness in reads from overloaded etcd servers (see [GitHub issue](https://github.com/etcd-io/etcd/issues/12678)). In such cases, `kube-apiserver` tracing warns as below: ``` "List" url:/api/v1/pods,user-agent... (started: ...) (total time: 1.208s): Trace[...]: [1.208s] [1.204s] Writing http response done count:4346 ``` And etcd warns as follows: ``` etcdserver: read-only range request key:"/registry/pods/" range_end:"/registry/pods0" revision:... range_response_count:500 size:291984 took too long (723.099118ms) to execute ``` Our deep-dive into etcd heap profile uncovered major inefficiency in the server warning logger that had a redundant encoding operation only to compute the size of range responses with [`proto.Size`](https://github.com/gogo/protobuf/blob/v1.3.2/proto/table_marshal.go#L2913-L2931) call. As a result, **a large range query had up to 60% heap allocation overhead, thus causing out-of-memory crashes, or OOM, in overloaded etcd servers** (see *Figure 1*). We **optimized the protocol buffer message size operation** and as a result, **reduced etcd memory consumption up to 50% during peak usage** (see *Figure 2*). It was a set of small code changes, but for years, such performance achievements were invisible without extensive testing and workload simulation. See [chaochn47@ of Amazon Web Services's investigation](https://github.com/etcd-io/etcd/issues/12835) and [patch to replace `proto.Size` calls](https://github.com/etcd-io/etcd/pull/12871). _**Figure 1:** etcd heap usage during a slow request warn logging for computing response size of a protocol buffer message. 61% of the heap was allocated in the `proto.Size` call path that encodes all key-value pairs in the message to compute the size._ _**Figure 2:** etcd heap usage during range query before and after replacing the `proto.Size` call. Optimizing the `proto.Size` call in the etcd server reduced memory usage up to 50%._ The etcd 3.4 release made backend read transactions fully concurrent by copying transaction buffers rather than sharing between writes and concurrent reads (see [code change from 3.4 release](https://github.com/etcd-io/etcd/pull/10523)). However, such a buffering mechanism comes with unavoidable copy overhead and negatively impacted write-heavy transaction performance, as creating concurrent read transactions acquires a mutex lock which then blocks incoming write transactions. etcd 3.5 improvements further increase transaction concurrency. 1. If a transaction includes a `PUT` (update) operation, the transaction instead shares the transaction buffer between reads and writes (same behavior as 3.3) in order to avoid copying buffers. This transaction mode can be disabled via `etcd --experimental-txn-mode-write-with-shared-buffer=false`. The benchmark results show that the **transaction throughput with a high write ratio has increased up to 2.7 times by avoiding copying buffers when creating a write transaction** (see *Figures 3* and *4*). This **benefits all kube-apiserver create and update calls that use etcd transactions** (see [etcd3 store v1.21 code](https://github.com/kubernetes/kubernetes/blob/v1.21.0/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go#L394-L401)). For more information, see [wilsonwang371@ of ByteDance's code change and benchmark results](https://github.com/etcd-io/etcd/pull/12896). _**Figure 3:** etcd transaction ratio with a high write ratio. The value at the top is the ratio of reads and writes. The first ratio, 0.125, is 1 read per 8 writes. The second ratio, 0.25, is 1 read for 4 writes. The value at the right bar represents the inverse ratio of transaction throughput before and after [etcd/pull/12896](https://github.com/etcd-io/etcd/pull/12896). With the shared buffer approach for writes, the transaction throughput is increased up to 2.7 times._ _**Figure 4:** etcd transaction ratio with a high read ratio. The value at the top is the ratio of reads and writes. The first ratio, 4.0, is 4 reads per 1 write. The second ratio, 8.0, is 8 reads per 1 write. The value at the right bar represents the inverse ratio of transaction throughput before and after [etcd/pull/12896](https://github.com/etcd-io/etcd/pull/12896). With the shared buffer approach for writes, the transaction throughput is increased up to 25%._ 1. etcd now caches the transaction buffer to avoid the unnecessary copy operations. This speeds up concurrent read transaction creation and as a result, the **transaction with a high read ratio has increased up to 2.4 times** (see *Figures 5* and *6*). See [wilsonwang371@ of ByteDance's code change and benchmark results](https://github.com/etcd-io/etcd/pull/12933). _**Figure 5:** etcd transaction ratio with a high write ratio. The value at the top is the ratio of reads and writes. The first ratio, 0.125, is 1 read per 8 writes. The second ratio, 0.25, is 1 read per 4 writes. The value at the right bar represents the inverse ratio of transaction throughput before and after [etcd/pull/12933](https://github.com/etcd-io/etcd/pull/12933). With the caching mechanism for read transactions, the transaction throughput is increased up to 1.4 times._ _**Figure 6:** etcd transaction ratio with a high read ratio. The value at the top is the ratio of reads and writes. The first ratio, 4.0, is 4 reads per 1 write. The second ratio, 8.0, is 8 reads per 1 write. The value at the right bar represents the inverse ratio of transaction throughput before and after [etcd/pull/12933](https://github.com/etcd-io/etcd/pull/12933). With the caching mechanism for read transactions, the transaction throughput is increased up to 2.5 times._ ## Monitoring Long-running load tests revealed that etcd server misrepresented its real memory usage by masking the impact of Go garbage collection. We discovered that etcd server with Go 1.12 changed the runtime to use `MADV_FREE` in Linux kernel, and as a result, reclaimed memory was not reflected in the resident set size, or RSS, metric. This had made the etcd memory usage metric inaccurately static, thus showing no sign of Go garbage collection. To fix this monitoring problem, we compile etcd 3.5 with Go 1.16 that defaults to `MADV_DONTNEED` on Linux. For more information, see *Figure 7* and [GitHub Go issue 42330](https://github.com/golang/go/issues/42330). _**Figure 7:** etcd memory usage during range query before and after setting `MADV_DONTNEED` in Go runtime. When run with `GODEBUG=madvdontneed=1"` or Go v1.16 or later, the etcd server accurately reports its memory usage in the CloudWatch `mem_used` metric or other monitoring tools such as `top`._ Monitoring is a fundamental service for reliability and observability. Monitoring enables individual service owners to understand the current state and identify possible causes for problem reports. Known as telemetry, the goal is to detect early warning signs and diagnose potential issues. etcd creates server logs with tracing information and publishes Prometheus metrics. This information helps us determine possible service impacts and causes. However, when a request call chain spans multiple external components (e.g., from `kube-apiserver` to etcd), it is harder to identify the issue. In order to efficiently identify the root cause, we added distributed tracing support using [OpenTelemetry](https://opentelemetry.io): When the **distributed tracing is enabled, etcd now uses OpenTelemetry to produce a trace across the RPC call chain and thus easily integrate with the surrounding ecosystem**. See *Figure 8*, [lilic@ of Red Hat's proposal](https://github.com/etcd-io/etcd/pull/12919), and [GitHub issue 12460](https://github.com/etcd-io/etcd/issues/12460). _**Figure 8:** Example jaeger tracing UI for etcd._ ## Testing Distributed systems are full of subtle edge cases. The fact that a bug may only emerge under special circumstances warrants exhaustive test coverage beyond simple unit testing. etcd runs integration, end-to-end, and failure injection tests, which provide a reliable and faster way to validate each change. However, as development continued with extended feature sets, flaky tests quickly piled up draining our productivity. So, we took on the series of arduous tasks that often required multiple hours of debugging to root cause of failures to improve test quality. Some notable changes include the following: * [reduce unit tests runtime by half](https://github.com/etcd-io/etcd/pull/12286) * [configure test logger](https://github.com/etcd-io/etcd/pull/12753) * [simplify test data cleanup](https://github.com/etcd-io/etcd/pull/12805) * [close gRPC servers after tests](https://github.com/etcd-io/etcd/pull/12782) ## Platforms etcd presubmit tests are now fast and reliable, but were mostly running on x86 processors. And there have been numerous requests to support other architectures like ARM (see [GitHub issue 12852](https://github.com/etcd-io/etcd/issues/12852)) and s390x (see [GitHub issue 11163](https://github.com/etcd-io/etcd/issues/11163)). A [self-hosted GitHub action runner](https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners) provides a consistent way of hosting various external test workers (see [GitHub issue 12856](https://github.com/etcd-io/etcd/issues/12856)). Using the GitHub action, etcd now [**runs tests on ARM-based AWS EC2 instances (Graviton)**](https://github.com/etcd-io/etcd/pull/12928), thereby [**officially supporting ARM64 (aarch64) platform**](https://github.com/etcd-io/etcd/pull/12929). In addition, we introduced a mechanism to support other platforms and **categorized support tiers based on testing coverage**. For more information, see the [Supported platforms documentation PR 273](https://github.com/etcd-io/website/pull/273) and the [Supported platforms documentation](https://github.com/etcd-io/website/pull/273). ## Developer experience To better support integration with external projects, **etcd now fully adopts the [Go 1.16 module](https://github.com/golang/go/wiki/Modules)**. This brought challenges as the existing codebase made migration arduous, causing [concerns regarding adoption in the community](https://github.com/etcd-io/etcd/issues/12484)). Tooling is an important part of etcd development and as a result, we needed a better solution to support our contributor experience. Using Go module enables clear separation between server and client code, ease of change management for dependency updates, and verifiable build system without a convoluted codebase for vendoring. With reproducible builds in place, we eliminated the need for vending dependency and as a result, we **reduced the etcd codebase size by half**. See [ptabor@ of Google's modularization proposal](https://docs.google.com/document/d/19UvKD7by_fEkzLMRi-QNSKEed4kYAW286S3DsNRPDOM/edit) and [code change to remove the `vendor` directory](https://github.com/etcd-io/etcd/pull/12279). To better isolate dependency trees, the etcd command-line interface now has a new administrative tool **`etcdutl` (not `etcdut-i-l`)**, and the sub-commands include **`etcdutl snapshot`** and **`etcdutl defrag`**: The **`etcdctl snapshot` and `etcdctl defrag` commands in previous releases are deprecated**. This change aligns well with the new Go module layouts: `etcdctl` solely relies on client v3 libraries, whereas `etcdutl` might depend on etcd server-side packages, such as `bolt` and backend database code. See [ptabor@ of Google's code change](https://github.com/etcd-io/etcd/pull/12971). To support [inclusive naming initiatives](https://inclusivenaming.org), etcd projects renamed the default branch `master` to `main` (see [`etcd-io/etcd` change](https://github.com/etcd-io/etcd/pull/12956) and [`etcd-io/website` change](https://github.com/etcd-io/website/issues/149)). The migration was seamless as GitHub handles the necessary redirects (see [GitHub renaming](https://github.com/github/renaming)). Each etcd write incurs an append message in Raft thus fsynced to disk. However, such persistence might not be desirable for testing. To work around this, we've added the `etcd --unsafe-no-fsync` flag to bypass disk writes for Raft WAL entries. See [crawshaw@ of Tailscale's code change](https://github.com/etcd-io/etcd/pull/11946) and [GitHub issue 11930](https://github.com/etcd-io/etcd/issues/11930). ## Community The diversity of etcd end users keeps expanding: Cloudflare [relies on etcd for managing its data center](https://blog.cloudflare.com/a-byzantine-failure-in-the-real-world), Grafana Cortex [stores its configuration data in etcd](https://grafana.com/blog/2020/04/07/how-a-production-outage-in-grafana-clouds-hosted-prometheus-service-was-caused-by-a-bad-etcd-client-setup), Netflix [Titus](https://netflix.github.io/titus/) uses etcd for [managing its container workloads](https://www.slideshare.net/aspyker/herding-kats-netflixs-journey-to-kubernetes-public), and Tailscale [runs its control plane on top of etcd](https://tailscale.com/blog/an-unlikely-database-migration). We have also extended our team of vendor contributors. In the etcd 3.5 release, we've [added two core maintainers](https://github.com/etcd-io/etcd/pull/12624); [Wenjia Zhang](https://github.com/wenjiaswe) of Google, who's been leading etcd community meetings and Kubernetes integration, and [Piotr Tabor](https://github.com/ptabor) of Google, who's been leading numerous bug fixes and codebase modularization work. The diversity of contributors is key to building a sustainable, welcoming open source project and fostering manageable work environments. See the [2021 CNCF etcd project journey report](https://www.cncf.io/cncf-etcd-project-journey-report) for more information. ## New etcd.io Since etcd joined [Cloud Native Computing Foundation](https://www.cncf.io) (CNCF) in [December 2018](https://www.cncf.io/blog/2018/12/11/cncf-to-host-etcd/), we have refactored all user-facing documentation into a dedicated repository [etcd-io/website](https://github.com/etcd-io/website) and [modernized its website hosting with Hugo](https://github.com/etcd-io/website/pull/244)). The migration was a huge undertaking that required multiple months of engineering efforts and communication between maintainers. The very blog post you are reading now is hosted on the new [etcd.io](https://etcd.io/blog), thanks to [lucperkins@](https://github.com/lucperkins), [chalin@](https://github.com/chalin) of CNCF, [nate-double-u@](https://github.com/nate-double-u) of CNCF, and many other community contributors. ## Future roadmaps Traffic overloads can cause cascading node failures and as a result, scaling such clusters becomes challenging and could impair the ability to recover from quorum loss. With so many mission critical systems built on top of etcd, defending etcd against overload is paramount. We will revisit the **etcd throttle feature to shed excessive loads gracefully**. Currently, the etcd project has two pending rate-limiter proposals: [vivekpatani@ of Apple's proposal](https://docs.google.com/document/d/1wQQ_L3cLyI1t14zp-PvarbjJYtlRsz_P9LwDR3uelc8/edit#heading=h.jofsq6eav4x7) and [tangcong@ of Tencent's proposal](https://github.com/etcd-io/etcd/pull/12290). Large range querying from `kube-apiserver` is still the most challenging source of process crashing, as it is relatively unpredictable. Our heap profile on such workloads found that **the etcd range request handler decodes and holds the entire response before sending it out to gRPC server, adding up to 37% heap allocation**. See *Figure 9* and [chaochn47@ of Amazon Web Services's investigation](https://github.com/etcd-io/etcd/issues/12835). Paginating range calls in client code doesn't fully address the issue, because it entails additional consistency considerations and still requires full relists for expired resources. For more information, see [kube-apiserver v1.21 code](https://github.com/kubernetes/kubernetes/blob/v1.21.0/staging/src/k8s.io/client-go/tools/cache/reflector.go#L302-L312)). To work around this inefficiency, **etcd needs to support range streams**. We will revisit [yangxuanjia@ of JD's range stream proposal](https://github.com/etcd-io/etcd/pull/12343), as it requires a significant level of effort to introduce such semantic changes in etcd and in downstream projects. _**Figure 9:** etcd usage during a range query for listing Kubernetes pods. 37% of the heap was allocated in etcd mvcc `rangeKeys` to hold key-value pairs for creating a range query response._ In order to reduce maintenance overhead, we are **completely deprecating the etcd v2 API in favor of a more performant and widely adopted v3 API**. The v2 storage translation layer via `etcd --experimental-enable-v2v3` remains experimental in 3.5 and to be removed in the next release. For details, see [ptabor@ of Google's proposal](https://github.com/etcd-io/etcd/issues/12913). Historically, etcd releases have been a large undertaking due to infrequency, because of the large delta, and a need for release automation. We will **develop an automated release system** that is more accessible to the community. [@etcdio]: https://twitter.com/etcdio [3.4-blog]: https://kubernetes.io/blog/2019/08/30/announcing-etcd-3-4/ [CHANGELOG 3.5]: https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.5.md [etcd 3.5]: https://github.com/etcd-io/etcd/releases/tag/v3.5.0 [Install]: /docs/v3.5/install/ --- ## File: content/en/blog/2022/etcd-integrates-continuous-fuzzing.md --- title: etcd Integrates Continuous Fuzzing spelling: cSpell:ignore Gyuho author: "[Adam Korczynski](https://twitter.com/AdamKorcz4), [David Korczynski](https://twitter.com/Davkorcz), [Sahdev Zala](https://twitter.com/sp_zala)" date: 2022-03-11 draft: false --- In the last few months, the team at [Ada Logics](https://adalogics.com) has worked on integrating continuous fuzzing into the etcd project. This was an effort focused on improving the security posture of etcd and ensuring a continued good experience for etcds users. The fuzzing integration involved enrolling etcd in the OSS-Fuzz project and writing a set of fuzzers that would bring the test coverage of etcd up to a mature level. In total, 18 fuzzers were written, and eight bugs were found, demonstrating the work’s value for etcd both short term and long term. All fuzzers were implemented by way of go-fuzz and when running in OSS-Fuzz instrumented by way of libFuzzer, and as such, etcd uses state-of-the-art open source fuzzing capabilities. The full report of the engagement can be found [here](https://github.com/etcd-io/etcd/blob/main/security/FUZZING_AUDIT_2022.PDF). The etcd project was created at CoreOS in 2013 and later joined the CNCF in 2018, from which it [graduated](https://www.cncf.io/announcements/2020/11/24/cloud-native-computing-foundation-announces-etcd-graduation/) in 2020. It is an open source, strongly consistent, distributed key-value store to reliably store data that a distributed system or cluster of machines needs to be accessed. It also provides highly desirable features like Watches to monitor changes to keys. etcd is a critical component of Kubernetes where it is used as the primary data store for cluster data such as the clusters state data and its data related to its desired state. Besides being a key component in Kubernetes, etcd is also used by [many other distributed systems](https://etcd.io/docs/v3.5/integrations/#projects-using-etcd). Because of its wide usage, etcd is a critical part of the open source ecosystem to fuzz for reliability bugs and security vulnerabilities. [The CNCF annual survey of 2021](https://www.cncf.io/reports/cncf-annual-survey-2021) found that 96% of companies are either using or evaluating Kubernetes, and etcd’s performance and security are important to continued business operations of these users. ## What is fuzzing? Fuzzing is a technique used to automate parts of the software testing process by way of a form of stress testing. The key idea is to write a fuzzing harness similar to a unit—or integration—test that will execute the application under test with some arbitrary input. The fuzzing engine that will run the fuzzing harness then uses genetic algorithms to extrapolate inputs that will cause the code under test to execute uniquely, i.e., generate inputs that trigger new code execution paths. The goal is then to observe if the code under test misbehaves in the event of any of the generated inputs. Fuzzing has been effective in uncovering reliability bugs and vulnerabilities in software for more than two decades, and open source software is increasingly adopting the technique. ## Etcd fuzzing overview In this engagement, the goal was to write a set of fuzzers that would cover a lot of the etcd codebase and integrate the setup into the open source fuzzing service OSS-Fuzz. OSS-Fuzz is a free service offered by Google for critical open source projects to run their fuzzers continuously and report any crashes. Continuous analysis is important due to fuzzing relying on genetic algorithms, which effectively means the fuzzers will improve over time, and OSS-Fuzz will run the fuzzers daily indefinitely. In addition to this, continuous analysis is crucial for capturing any regressions. Etcd is written in the Go programming language, making it safe from memory-corruptions. Fuzzing Go will find panics such as slice/index out of range, nil-pointer dereferences, invalid type assertions, timeouts, out of memory. At the end of this engagement, eight issues were found, all of which were fixed. They are broken down as such: At the end of this engagement, the fuzzers provide significant coverage of the etcd project, including critical parts such as the etcd server, WAL, the auth store, and the raft package. During the engagement, Ada Logics found that only a few of the critical parts of etcd would be accessible with a byte slice or string but instead accepted complex types such as structs. An example of this is [the fuzzers for the etcd server](https://github.com/cncf/cncf-fuzzing/blob/main/projects/etcd/etcdserver_fuzzer.go), which configures and sets up an etcd server and then creates a series of pseudo-randomized structs representing different requests sent to the server. To write these fuzzers, Ada Logics used [go-fuzz-headers](https://github.com/AdaLogics/go-fuzz-headers) to deterministically create pseudo-random structs from the data provided by libFuzzer. ## Closing thoughts The etcd team is thankful to CNCF and Chris Aniszczyk for providing the opportunity to work with Ada Logics to develop new fuzzers for etcd. The software security is taken seriously by the CNCF, and it earlier funded the etcd project for a [third-party security audit](https://www.cncf.io/blog/2020/08/05/etcd-security-audit/). We also want to thank all the etcd maintainers and reviewers, especially Marek Siarkowicz, Piotr Tabor, and Benjamin Wang, for their quick reviews of the fixes. The fuzzing findings and fixes are valuable add-ons to the previous conclusions of the security audit. etcd project has efficient test suites, and code changes are backed by tests, but the newly developed fuzzers and findings have provided significant value to the project. During the fuzzing, only eight issues were found, which revalidated the high quality of the etcd code. The etcd team should maintain the newly developed fuzzers and build on them to continue code quality and security. --- ## File: content/en/blog/2023/engage_with_etcd_at_kccncna2023.md --- title: Engage with the etcd project at KubeCon NA 2023 author: "[James Blair](https://github.com/jmhbnz), Red Hat" date: 2023-11-05 draft: false --- KubeCon NA 2023 in Chicago is just around the corner! This year, the etcd project has a diverse range of talks, tutorials, and even interactive contribfest sessions for you to get involved in . As a critical foundational pillar of the Kubernetes ecosystem, etcd's presence at Kubecon underscores its importance in ensuring all our Kubernetes clusters continue to have robust and reliable distributed persistent state. Here's a detailed overview of what you can expect from the Etcd Project's presence at KubeCon NA 2023: ## 1. Schedule of talks Below is a list of talks on etcd, you can add these to your custom agenda by clicking the links for each talk through to . | **Talk Title** | **Speakers** | **Date & Time (CST)** | |----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|--------------------------------| | **[Journey Through Time: Understanding Etcd Revisions and Resource Versions in Kubernetes](https://kccncna2023.sched.com/event/1R2m8/journey-through-time-understanding-etcd-revisions-and-resource-versions-in-kubernetes-priyanka-saggu-suse)** | Priyanka Saggu, SUSE | Tuesday, November 7, 11:00am | | **[Mastering Etcd Observability: A Comprehensive Guide to Metrics, Monitoring, and Incident Handling](https://kccncna2023.sched.com/event/1R2rD/tutorial-mastering-etcd-observability-a-comprehensive-guide-to-metrics-monitoring-and-incident-handling-bogdan-kanivets-vivek-patani-apple)** | Bogdan Kanivets & Vivek Patani, Apple | Wednesday, November 8, 2:30pm | | **[Forging a Stronger Bond Between Etcd and Kubernetes](https://kccncna2023.sched.com/event/1R2rt/forging-a-stronger-bond-between-etcd-and-kubernetes-marek-siarkowicz-wenjia-zhang-google-james-blair-red-hat)** | Marek Siarkowicz & Wenjia Zhang, Google; James Blair, Red Hat | Wednesday, November 8, 3:25pm | | **[Insights and Gotchas from the Zero-Downtime Migration of 10000+ Cloud Hosted Etcd Key-Value Stores](https://kccncna2023.sched.com/event/1R2tG/insights-and-gotchas-from-the-zero-downtime-migration-of-10000-cloud-hosted-etcd-key-value-stores-prabhakar-palanivel-oracle-corporation)** | Prabhakar Palanivel, Oracle Corporation | Thursday, November 9, 11:00am | | **[Secrets of Running Etcd](https://kccncna2023.sched.com/event/1R2vl/secrets-of-running-etcd-marek-siarkowicz-google)** | Marek Siarkowicz, Google | Thursday, November 9, 4:00pm | ## 2. Contribfest - Learn from the maintainers and get involved Join the etcd contributor community at this [hands on session](https://kccncna2023.sched.com/event/1R2p2/contribfest-etcd-learn-from-the-maintainers-and-get-involved-marek-siarkowicz-wenjia-zhang-google-james-blair-josh-berkus-red-hat). We'll be working on improving key features and testing for Etcd, and in the process we’ll teach those new to the project how to contribute. Etcd welcomes both new contributors and those who want to “level up”. Attendees should be familiar with programming in Go, using GitHub, and should bring a laptop on which they can do cloud-native development: either a Linux laptop or your own Github Devcontainer setup. - **Facillitators:** Marek Siarkowicz & Wenjia Zhang, Google; James Blair & Josh Berkus, Red Hat - **Date & Time:** Tuesday, November 7, 4:30pm CST ## 3. Project pavilion booth: Meet the etcd team Along with the engaging sessions above, the Etcd Project will also be staffing a booth in the project pavilion. This is a fantastic opportunity for you to interact directly with the etcd team, ask questions, and learn more about the project's latest developments. Whether you are a beginner exploring the world of Kubernetes or an experienced developer seeking insights into etcd's intricacies, the etcd project's booth is the place to stop by, say hello and ask any questions you might have. ## 4. SIG-etcd meet and greet Lastly, on Thursday, November 9 from 12:00pm - 3:00pm the SIG-etcd team will have a table at the [Meet the Kubernetes Contributor Community](https://github.com/kubernetes/community/issues/7541) event. If you have questions specifically about the new etcd kubernetes special interest group please stop by. ## We hope to see you there Don't miss these opportunities to enhance your understanding of etcd and the pivotal role it plays in our Kubernetes ecosystem. Join us at KubeCon NA 2023 in Chicago, and let's explore the possibilities of etcd together! --- *Note: Schedule and session details are subject to change. Please refer to the [official KubeCon NA 2023 schedule](https://kccncna2023.sched.com) for the most up-to-date information.* --- ## File: content/en/blog/2023/how_to_debug_large_db_size_issue.md --- title: How to debug large db size issue? author: "[Benjamin Wang](https://github.com/ahrtr), VMware" date: 2023-01-04 draft: false --- # Background Users can configure the quota of the backend db size using flag `--quota-backend-bytes`. It's the max number of bytes the etcd db file may consume, namely the ${etcd-data-dir}/member/snap/db file. Its default value is 2GB, and the suggested max value is 8GB. 2GB is usually sufficient for most use cases. If you run out of the db quota, you will see error message `etcdserver: mvcc: database space exceeded` when trying to write more data, and see alarm "NOSPACE" (see example below) when checking the endpoint status or health state. It would be better to figure out whether it's expected. It's exactly the reason why I provide this guide. ``` $ etcdctl endpoint status -w table +----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+ | 127.0.0.1:2379 | 8e9e05c52164694d | 3.5.5 | 25 kB | true | false | 2 | 5 | 5 | memberID:10276657743932975437 | | | | | | | | | | | alarm:NOSPACE | +----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+ $ etcdctl endpoint health -w table +----------------+--------+------------+---------------------------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------+--------+------------+---------------------------+ | 127.0.0.1:2379 | false | 1.850456ms | Active Alarm(s): NOSPACE | +----------------+--------+------------+---------------------------+ ``` # How to check db size The easiest way is to execute `ls -lrt ${etcd-data-dir}/member/snap` command directly. For example, the db file is 5148672 bytes in the following example, ``` # ls -lrt /path-2-db-data-dir/member/snap/ total 5108 -rw-r--r-- 1 vcap vcap 13503 Nov 15 00:48 000000000000001f-0000000000c0433e.snap -rw-r--r-- 1 vcap vcap 13503 Nov 15 12:33 000000000000001f-0000000000c1c9df.snap -rw-r--r-- 1 vcap vcap 13502 Nov 16 00:17 000000000000001f-0000000000c35080.snap -rw-r--r-- 1 vcap vcap 13503 Nov 16 12:03 000000000000001f-0000000000c4d721.snap -rw-r--r-- 1 vcap vcap 13502 Nov 16 23:52 0000000000000023-0000000000c65dc2.snap -rw------- 1 vcap vcap 5148672 Nov 17 02:44 db ``` The second way to check db size is to execute `etcdctl endpoint status` command. For example, it's about 5.1MB in the following example. Note that if you do not set the `--cluster` flag, then you need to get all members' endpoints included in `--endpoints`, e.g, `--endpoints https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379`. ``` # etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key --endpoints https://etcd-0:2379 endpoint status -w table --cluster +------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://master-0.etcd.cfcr.internal:2379 | 17f206fd866fdab2 | 3.5.4 | 5.2 MB | false | false | 35 | 13024257 | 13024257 | | | https://master-2.etcd.cfcr.internal:2379 | 604ea1193b383592 | 3.5.4 | 5.1 MB | false | false | 35 | 13024257 | 13024257 | | | https://master-1.etcd.cfcr.internal:2379 | 9dccb73515ee278f | 3.5.4 | 5.1 MB | true | false | 35 | 13024257 | 13024257 | | +------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ ``` You can also display the output using json format (See example below). The benefit is that you can get the both `dbSize` and `dbSizeInUse`. Let's use the first endpoint as an example, its `dbSize` is 5177344 bytes, and `dbSizeInUse` is 2039808 bytes. **It means that there are (5177344-2039808) bytes of free space, which can be reclaimed by executing the defragmentation operation**. ``` # etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key --endpoints https://etcd-0:2379 endpoint status -w json --cluster [ { "Endpoint": "https://master-0.etcd.cfcr.internal:2379", "Status": { "header": { "cluster_id": 7895810959607866000, "member_id": 1725449293188291300, "revision": 9662972, "raft_term": 35 }, "version": "3.5.4", "dbSize": 5177344, "leader": 11370664597832739000, "raftIndex": 13024242, "raftTerm": 35, "raftAppliedIndex": 13024242, "dbSizeInUse": 2039808 } }, { "Endpoint": "https://master-2.etcd.cfcr.internal:2379", "Status": { "header": { "cluster_id": 7895810959607866000, "member_id": 6939661205564306000, "revision": 9662972, "raft_term": 35 }, "version": "3.5.4", "dbSize": 5148672, "leader": 11370664597832739000, "raftIndex": 13024242, "raftTerm": 35, "raftAppliedIndex": 13024242, "dbSizeInUse": 2031616 } }, { "Endpoint": "https://master-1.etcd.cfcr.internal:2379", "Status": { "header": { "cluster_id": 7895810959607866000, "member_id": 11370664597832739000, "revision": 9662972, "raft_term": 35 }, "version": "3.5.4", "dbSize": 5144576, "leader": 11370664597832739000, "raftIndex": 13024242, "raftTerm": 35, "raftAppliedIndex": 13024242, "dbSizeInUse": 2039808 } } ] ``` etcd also exposes the following metrics (click [here](https://etcd.io/docs/current/metrics/etcd-metrics-latest.txt) to see a full list of metrics), | Metrics | Description | |-------|--------------| | etcd_server_quota_backend_bytes | Current backend storage quota size in bytes. | | etcd_mvcc_db_total_size_in_bytes | Total size of the underlying database physically allocated in bytes. | | etcd_mvcc_db_total_size_in_use_in_bytes | Total size of the underlying database logically in use in bytes. | # Compaction & Defragmentation etcd supports MVCC(Multi-Version Concurrent Control), and it keeps an exact history of its key spaces. The compaction operation is the only way to purge history. But the free space will not be reclaimed automatically, instead you should perform defragmentation to reclaim the free space after the compaction operation. So usually Compaction + Defragmentation is the right way to purge history and reclaim unused storage space. Please follow guide [maintenance/#space-quota](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) to manually perform compaction and defragmentation operations. Notes: 1. Compaction is a cluster-wide operation, so you only need to execute compaction once on whichever etcd member. Of course, it will not do any harm if executing it multiple times. 2. Defragmentation is a time-consuming task, so it's recommended to do it for each member one by one. 3. Defragmentation is an expensive operation, so you should do it as infrequent as possible. On the other hand, you also need to make sure any etcd member will not run out of the storage quota. `etcdctl defrag [flags]` can be used for defragmentation, but tools such as [etcd-defrag](https://github.com/ahrtr/etcd-defrag/) can provide enhanced functionality and ease of use and should be considered to use. 4. **There is a known issue that etcd might run into data inconsistency issue if it crashes in the middle of an online defragmentation operation using `etcdctl` or clientv3 API. All the existing v3.5 releases are affected, including 3.5.0 ~ 3.5.5. So please use `etcdutl` to offline perform defragmentation operation**, but this requires taking each member offline one at a time. It means that you need to stop each etcd instance firstly, then perform defragmentation using `etcdutl`, start the instance at last. Please refer to the issue 1 in [public statement](https://groups.google.com/g/etcd-dev/c/8S7u6NqW6C4/m/4Z84zCuIAQAJ). 5. Please run `etcdctl alarm disarm` if there is a `NOSPACE` alarm. The following example shows you how to execute defragmention using `etcdutl`, ``` $ etcdutl defrag --data-dir ~/tmp/etcd/infra1.etcd/ ``` # What data occupies most of the storage space If the compaction + defragmentation doesn't help; in other words, if the db size is still exceeding or close to the quota, then you need to figure out what data is consuming most of the storage space. The straightforward way is to count the number of each kind of object. If the etcd cluster is supporting Kubernetes apiserver, then execute command below to do the statistics. In the following example, there are about 971K events, so obvious the events occupy most of the storage space. **The next step is to use your best judgement to figure out the root cause**. Is it expected? Have you or your user change the value of `--event-ttl` (apiserver flag, defaults to 1h)? Is it running into a known issue (e.g. [107170](https://github.com/kubernetes/kubernetes/issues/107170) )? ``` # etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key --endpoints https://etcd-0:2379 get /registry --prefix --keys-only | grep -v ^$ | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr 971223 events 51011 pods 10008 replicasets 10008 deployments 3886 services 2555 secrets 2549 serviceaccounts 2514 configmaps 2507 namespaces 1947 endpointslices 506 leases 500 minions 500 csinodes 500 cns.vmware.com 77 clusterroles 64 clusterrolebindings 34 apiregistration.k8s.io 13 podsecuritypolicy 12 rolebindings 12 flowschemas 11 apiextensions.k8s.io 10 roles 8 prioritylevelconfigurations 3 masterleases 2 vmware.com 2 ranges 2 priorityclasses 2 daemonsets 2 controllerrevisions 1 validatingwebhookconfigurations 1 nsx.vmware.com 1 jobs 1 csidrivers ``` Usually when there are huge entries, it may take a long time to execute command above, so please set larger timeout values (see below), otherwise you may run into "context deadline exceeded". ``` --dial-timeout 10s --command-timeout 20s ``` If the etcd instance is not running, then you can use [etcd-dump-db](https://github.com/etcd-io/etcd/tree/main/tools/etcd-dump-db) to do the similar analysis(see example below). Note that you can build the binary `etcd-dump-db` using command `go build` with a golang version 1.19.x. ``` ./etcd-dump-db iterate-bucket /path-2-etcd-data-dir/member/snap/db key --decode | egrep -o '"/registry.*' | cut -d'|' -f1 | grep -v ^$ | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr 630 leases 150 masterleases 82 nsx.vmware.com 77 clusterroles 64 clusterrolebindings 54 secrets 48 serviceaccounts 33 apiregistration.k8s.io 18 pods 13 podsecuritypolicy 12 rolebindings 12 flowschemas 12 configmaps 10 services 10 roles 9 apiextensions.k8s.io 8 replicasets 8 prioritylevelconfigurations 8 minions 8 deployments 6 namespaces 5 endpointslices 3 csinodes 2 ranges 2 priorityclasses 2 daemonsets 2 controllerrevisions 1 validatingwebhookconfigurations 1 jobs ``` # Solutions If the behavior (db size exceeds the quota) is expected, then you can set a bigger value for `--quota-backend-bytes`. You need to make sure your cloud provider supports this, otherwise the manual update might not survive across cluster upgrading or VM recreating. **Note that etcd (actually boltDB) maps the db file into memory directly, so a larger value also means more memory usage**. Just I mentioned in the beginning of this post, the suggested max value is 8GB. Of course, If your VM has big memory (e.g. 64GB), it's OK to set a value > 8GB. The other solution is to set per-resource etcd servers overrides using apiserver flag `--etcd-servers-overrides`. In the following example, there are two etcd clusters; one for the normal usage, and the other dedicated to events. ``` --etcd-servers="https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379" \ --etcd-servers-overrides="/events#https://etcd-3:2379,https://etcd-4:2379,https://etcd-5:2379" ``` As a mitigation, you may also want to delete unneeded objects to free up some space. You will need to delete objects before running compaction and defragmentation to free up the space. If the behavior (db size exceeds the quota) isn't expected, then you'd better figure out the root cause and resolve it firstly. If you insist on applying solutions mentioned above, it can mitigate the issue instead of resolving it. --- ## File: content/en/blog/2023/introducing-sig-etcd.md --- title: Introducing sig-etcd author: "[Han Kang](https://github.com/logicalhan), [Marek Siarkowicz](https://github.com/serathius), Frederico Muñoz" date: 2023-11-01 draft: false --- Special Interest Groups (SIGs) are a fundamental part of the Kubernetes project, with a substantial share of the community activity happening within them. When the need arises, [new SIGs can be created](https://github.com/kubernetes/community/blob/master/sig-wg-lifecycle.md), and that was precisely what happened recently. [SIG etcd](https://github.com/kubernetes/community/blob/master/sig-etcd/README.md) is the most recent addition to the list of Kubernetes SIGs. In this article we will get to know it a bit better, understand its origins, scope, and plans. ## The critical role of etcd If we look inside the control plane of a Kubernetes cluster, we will find [etcd](https://kubernetes.io/docs/concepts/overview/components/#etcd), a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data -- this description alone highlights the critical role that etcd plays, and the importance of it within the Kubernetes ecosystem. This critical role makes the health of the etcd project and community an important consideration, and [concerns about the state of the project](https://groups.google.com/a/kubernetes.io/g/steering/c/e-O-tVSCJOk/m/N9IkiWLEAgAJ) in early 2022 did not go unnoticed. The changes in the maintainer team, amongst other factors, contributed to a situation that needed to be addressed. ## Why a special interest group With the critical role of etcd in mind, it was proposed that the way forward would be to create a new special interest group. If etcd was already at the heart of Kubernetes, creating a dedicated SIG not only recognises that role, it would make etcd a first-class citizen of the Kubernetes community. Establishing SIG etcd creates a dedicated space to make explicit the contract between etcd and Kubernetes api machinery and to prevent, on the etcd level, changes which violate this contract. Additionally, etcd will be able to adopt the processes that Kubernetes offers its SIGs ([KEPs](https://www.kubernetes.dev/resources/keps/), [PRR](https://github.com/kubernetes/community/blob/master/sig-architecture/production-readiness.md), [phased feature gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/), amongst others) in order to improve the consistency and reliability of the codebase. Being able to use these processes will be a substantial benefit to the etcd community. As a SIG, etcd will also be able to draw contributor support from Kubernetes proper: active contributions to etcd from Kubernetes maintainers would decrease the likelihood of breaking Kubernetes changes, through the increased number of potential reviewers and the integration with existing testing framework. This will not only benefit Kubernetes, which will be able to better participate and shape the direction of etcd in terms of the critical role it plays, but also etcd as a whole. ## About SIG etcd The recently created SIG is already working towards its goals, defined in its [Charter](https://github.com/kubernetes/community/blob/master/sig-etcd/charter.md) and [Vision](https://github.com/kubernetes/community/blob/master/sig-etcd/vision.md). The purpose is clear: to ensure etcd is a reliable, simple, and scalable production-ready store for building cloud-native distributed systems and managing cloud-native infrastructure via orchestrators like Kubernetes. The scope of SIG etcd is not exclusively about etcd as a Kubernetes component, it also covers etcd as a standard solution. Our goal is to make etcd the most reliable key-value storage to be used anywhere, unconstrained by any Kubernetes-specific limits and scaling to meet the requirements of many diverse use-cases. We are confident that the creation of SIG etcd constitutes an important milestone in the lifecycle of the project, simultaneously improving etcd itself, and also the integration of etcd with Kubernetes. We invite everyone interested in etcd to [visit our page](https://github.com/kubernetes/community/blob/master/sig-etcd/README.md), [join us at our Slack channel](https://kubernetes.slack.com/messages/etcd), and get involved in this new stage of etcd's life.