OWNERS Documentation

1. Project Overview & Quickstart (etcd-io/OWNERS) Copy Chapter

OWNERS

Open-source repository etcd-io/OWNERS

Repository Details Copy Section

• Repository: etcd-io/OWNERS
• Primary Language: Code

Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.

2. Official Technical Reference & Guides (etcd-io/website) Copy Chapter

File: README.md Copy Chapter

etcd.io

The etcd.io website, built using Hugo and hosted on Netlify.

Cloud build Copy Chapter

Visit https://gitpod.io/#https://github.com/etcd-io/website to launch a Gitpod.io IDE that will allow you to build, preview and make changes to this repo.

Local build Copy Chapter

To build and serve the site, you'll need the latest LTS release of Node.
Like Netlify, we use nvm, the Node Version Manager, to install and
manage Node versions:

$ nvm install --lts

Setup Copy Section

1. Clone this repo.
2. From a terminal window, change to the cloned repo directory.
3. Get NPM packages and git submodules, including the the Docsy theme:
   console Copy

   $ npm install

Build or serve the site Copy Section

To locally serve the site at localhost:8888, run the following command:

$ npm run serve

NOTE: If you want to check links locally, you (temporarily) might have
some extra setup to do. For details, see issue
#459.

To build and check links, run these commands:

$ npm run build
$ npm run check-links

Contributing Copy Chapter

We welcome issues and PRs! For details, see Contributing.

If you submit a PR, Netlify will automatically create a deploy preview so
that you can view your changes. Once merged, Netlify automatically deploys to
the production site etcd.io.

Releasing a new version of the etcd documentation Copy Chapter

IMPORTANT: this section is out-of-date, see issue #406.

Follow these steps to add documentation for a newly released version of etcd, vX.Y:

• Recursively copy content/docs/next into
  content/docs/vX.Y, where vX.Y is the newly released version of etcd. For example:

  bash Copy

  cp -r content/en/docs/next content/en/docs/v3.6

• In the _index.md file at the root of the new directory, update the frontmatter
  to reflect the new version:

  text Copy

  ---
  title: etcd version X.Y
  weight: 1000
  cascade:
    version: vX.Y
  ---

• Add the version to the params.versions.all array in the
  config.toml configuration file.

• If the version is meant to be the latest version of etcd, change the
  params.latest_stable_version parameter to the desired new version.

• Submit a pull request with the changes.

File: content/en/blog/2014/etcd-freeBSD.md Copy Chapter

title: FreeBSD date: 2014-01-23 Copy Chapter

This is an adaptation of a page previously found in the Platforms section of the documentation which described etcd deployments on various platform services.
The original page was authored by Bartłomiej Rutkowski and others.

Starting with version 0.1.2 both etcd and etcdctl have been ported to FreeBSD and can be installed either via packages or ports system. Their versions have been recently updated to 0.2.0 so now etcd and etcdctl can be enjoyed on FreeBSD 10.0 (RC4 as of now) and 9.x, where they have been tested. They might also work when installed from ports on earlier versions of FreeBSD, but it is untested; caveat emptor.

Installation Copy Chapter

Using pkgng package system Copy Section

1. If pkg­ng is not installed, install it with command pkg and answering 'Y' when asked.

2. Update the repository data with pkg update.

3. Install etcd with pkg install coreos-etcd coreos-etcdctl.

4. Verify successful installation by confirming pkg info | grep etcd matches:

r@fbsd­10:/ # pkg info | grep etcd
coreos­etcd­0.2.0              Highly­available key value store and service discovery
coreos­etcdctl­0.2.0           Simple commandline client for etcd
r@fbsd­10:/ #

5. etcd and etcdctl are ready to use! For more information about using pkgng, please see: http://www.freebsd.org/doc/handbook/pkgng­intro.html

Using ports system Copy Section

1. If ports is not installed, install with portsnap fetch extract (it may take some time depending on hardware and network connection).

2. Build etcd with cd /usr/ports/devel/etcd && make install clean. There will be an option to build and install documentation and etcdctl with it.

3. If etcd wasn't installed with etcdctl, it can be built later with cd /usr/ports/devel/etcdctl && make install clean.

4. Verify successful installation by confirming pkg info | grep etcd matches:

r@fbsd­10:/ # pkg info | grep etcd
coreos­etcd­0.2.0              Highly­available key value store and service discovery
coreos­etcdctl­0.2.0           Simple commandline client for etcd
r@fbsd­10:/ #

5. etcd and etcdctl are ready to use! For more information about using ports system, please see: https://www.freebsd.org/doc/handbook/ports­using.html

Issues Copy Chapter

If there are any issues with the build/install procedure or there's a problem that is local to FreeBSD only (for example, by not being able to reproduce it on any other platform, like OSX or Linux), please send a problem report using this page for more information: http://www.freebsd.org/send­pr.html

File: content/en/blog/2017/etcd-container-linux-with-systemd.md Copy Chapter

title: Container Linux with systemd date: 2017-05-23 Copy Chapter

This is an adaptation of a page previously found in the Platforms section of the documentation which described etcd deployments on various platform services.
The original page was authored by Gyuho Lee.

The following blog post shows how to run etcd with systemd under Container Linux.

Provisioning an etcd cluster Copy Chapter

Cluster bootstrapping in Container Linux is simplest with Ignition; coreos-metadata.service dynamically fetches the machine's IP for discovery. Note that etcd's discovery service protocol is only meant for bootstrapping, and cannot be used with runtime reconfiguration or cluster monitoring.

The Container Linux Config Transpiler compiles etcd configuration files into Ignition configuration files:

etcd:
  version: 3.2.0
  name: s1
  data_dir: /var/lib/etcd
  advertise_client_urls:       http://{PUBLIC_IPV4}:2379
  initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
  listen_client_urls:          http://0.0.0.0:2379
  listen_peer_urls:            http://{PRIVATE_IPV4}:2380
  discovery:                   https://discovery.etcd.io/<token>

ct would produce the following Ignition Config:

$ ct --platform=gce --in-file /tmp/ct-etcd.cnf
{"ignition":{"version":"2.0.0","config"...

{
  "ignition":{"version":"2.0.0","config":{}},
  "storage":{},
  "systemd":{
    "units":[{
      "name":"etcd-member.service",
      "enable":true,
      "dropins":[{
        "name":"20-clct-etcd-member.conf",
        "contents":"[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nEnvironment=\"ETCD_IMAGE_TAG=v3.1.8\"\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n  --name=\"s1\" \\\n  --data-dir=\"/var/lib/etcd\" \\\n  --listen-peer-urls=\"http://${COREOS_GCE_IP_LOCAL_0}:2380\" \\\n  --listen-client-urls=\"http://0.0.0.0:2379\" \\\n  --initial-advertise-peer-urls=\"http://${COREOS_GCE_IP_LOCAL_0}:2380\" \\\n  --advertise-client-urls=\"http://${COREOS_GCE_IP_EXTERNAL_0}:2379\" \\\n  --discovery=\"https://discovery.etcd.io/\u003ctoken\u003e\""}]}]},
      "networkd":{},
      "passwd":{}}

To avoid accidental misconfiguration, the transpiler helpfully verifies etcd configurations when generating Ignition files:

etcd:
  version: 3.2.0
  name: s1
  data_dir_x: /var/lib/etcd
  advertise_client_urls:       http://{PUBLIC_IPV4}:2379
  initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
  listen_client_urls:          http://0.0.0.0:2379
  listen_peer_urls:            http://{PRIVATE_IPV4}:2380
  discovery:                   https://discovery.etcd.io/<token>

$ ct --platform=gce --in-file /tmp/ct-etcd.cnf
warning at line 3, column 2
Config has unrecognized key: data_dir_x

See Container Linux Provisioning for more details.

etcd 3.x service Copy Chapter

Container Linux does not include etcd 3.x binaries by default. Different versions of etcd 3.x can be fetched via etcd-member.service.

Confirm unit file exists:

systemctl cat etcd-member.service

Check if the etcd service is running:

systemctl status etcd-member.service

Example systemd drop-in unit to override the default service settings:

cat > /tmp/20-cl-etcd-member.conf <<EOF
[Service]
Environment="ETCD_IMAGE_TAG=v3.2.0"
Environment="ETCD_DATA_DIR=/var/lib/etcd"
Environment="ETCD_SSL_DIR=/etc/ssl/certs"
Environment="ETCD_OPTS=--name s1 \
  --listen-client-urls https://10.240.0.1:2379 \
  --advertise-client-urls https://10.240.0.1:2379 \
  --listen-peer-urls https://10.240.0.1:2380 \
  --initial-advertise-peer-urls https://10.240.0.1:2380 \
  --initial-cluster s1=https://10.240.0.1:2380,s2=https://10.240.0.2:2380,s3=https://10.240.0.3:2380 \
  --initial-cluster-token mytoken \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file /etc/ssl/certs/etcd-root-ca.pem \
  --cert-file /etc/ssl/certs/s1.pem \
  --key-file /etc/ssl/certs/s1-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file /etc/ssl/certs/etcd-root-ca.pem \
  --peer-cert-file /etc/ssl/certs/s1.pem \
  --peer-key-file /etc/ssl/certs/s1-key.pem \
  --auto-compaction-retention 1"
EOF
mv /tmp/20-cl-etcd-member.conf /etc/systemd/system/etcd-member.service.d/20-cl-etcd-member.conf

Or use a Container Linux Config:

systemd:
  units:
    - name: etcd-member.service
      dropins:
        - name: conf1.conf
          contents: |
            [Service]
            Environment="ETCD_SSL_DIR=/etc/ssl/certs"

etcd:
  version: 3.2.0
  name: s1
  data_dir: /var/lib/etcd
  listen_client_urls:          https://0.0.0.0:2379
  advertise_client_urls:       https://{PUBLIC_IPV4}:2379
  listen_peer_urls:            https://{PRIVATE_IPV4}:2380
  initial_advertise_peer_urls: https://{PRIVATE_IPV4}:2380
  initial_cluster:             s1=https://{PRIVATE_IPV4}:2380,s2=https://10.240.0.2:2380,s3=https://10.240.0.3:2380
  initial_cluster_token:       mytoken
  initial_cluster_state:       new
  client_cert_auth:            true
  trusted_ca_file:             /etc/ssl/certs/etcd-root-ca.pem
  cert_file:                   /etc/ssl/certs/s1.pem
  key_file:                    /etc/ssl/certs/s1-key.pem
  peer_client_cert_auth:       true
  peer_trusted_ca_file:        /etc/ssl/certs/etcd-root-ca.pem
  peer_cert_file:              /etc/ssl/certs/s1.pem
  peer_key_file:               /etc/ssl/certs/s1-key.pem
  auto_compaction_retention:   1

$ ct --platform=gce --in-file /tmp/ct-etcd.cnf
{"ignition":{"version":"2.0.0","config"...

To see all runtime drop-in changes for system units:

systemd-delta --type=extended

To enable and start:

systemctl daemon-reload
systemctl enable --now etcd-member.service

To see the logs:

journalctl --unit etcd-member.service --lines 10

To stop and disable the service:

systemctl disable --now etcd-member.service

etcd 2.x service Copy Chapter

Container Linux includes a unit file etcd2.service for etcd 2.x, which will be removed in the near future. See Container Linux FAQ for more details.

Confirm unit file is installed:

systemctl cat etcd2.service

Check if the etcd service is running:

systemctl status etcd2.service

To stop and disable:

systemctl disable --now etcd2.service

File: content/en/blog/2017/etcd-deployments-on-AWS.md Copy Chapter

title: Deployment of etcd on AWS EC2 date: 2017-04-10 Copy Chapter

This is an adaptation of a page previously found in the Platforms section of the documentation which described etcd deployments on various platform services.
The original page was authored by Caleb Miles and others.

This post provides an introduction to design considerations when designing an etcd deployment on AWS EC2 and how AWS specific features may be utilized in that context.
Also, this post assumes operational knowledge of Amazon Web Services (AWS), specifically Amazon Elastic Compute Cloud (EC2).

Table of Contents

• Capacity Planning
• Cluster Design
• Availability
• Data durability after member failure
• Performance/Throughput
• Network
• Disk
• Self-healing
• Next Steps

Capacity Planning Copy Chapter

As a critical building block for distributed systems it is crucial to perform adequate capacity planning in order to support the intended cluster workload.
As etcd is a highly available and strongly consistent data store, so increasing the number of nodes in an etcd cluster will generally affect performance adversely.
This makes sense intuitively, as more nodes means more members for the leader to coordinate state across.
The most direct way to increase throughput and decrease latency of an etcd cluster is allocate more disk I/O, network I/O, CPU, and memory to the cluster members.
In this scenario, it is impossible to temporarily divert incoming requests to the cluster, though scaling the EC2 instances which comprise the etcd cluster members one at a time may improve performance.
It is, however, best to avoid bottlenecks through capacity planning.

The etcd team has produced a hardware recommendation guide which is very useful for "ballparking" how many nodes and what instance type are necessary for a cluster.

AWS provides a service for creating groups of EC2 instances which are dynamically sized to match load on the instances.
Using an Auto Scaling Group (ASG) to dynamically scale an etcd cluster, is not recommended for several reasons including the following:

• etcd performance is generally inversely proportional to the number of members in a cluster due to the synchronous replication which provides strong consistency of data stored in etcd.

• The operational complexity of adding lifecycle hooks to properly add and remove members from an etcd cluster by modifying the runtime configuration.

Auto Scaling Groups do provide a number of benefits besides cluster scaling which include:

• distribution of EC2 instances across Availability Zones (AZs)
• EC2 instance fail over across AZs
• consolidated monitoring and life cycle control of instances within an ASG

The use of an ASG to create a self healing etcd cluster is one of the design considerations when deploying an etcd cluster to AWS.

Cluster design Copy Chapter

The purpose of this section is to provide foundational guidance for deploying etcd on AWS. The discussion will be framed by the following three critical design criteria about the etcd cluster itself:

• Block device provider: Limited to the tradeoffs between EBS or instance storage (InstanceStore)
• Cluster topology: Number of nodes that should make up an etcd cluster and whether these nodes should be distributed over multiple AZs
• Managing etcd members: Creating a static cluster of EC2 instances or using an ASG.

The intended cluster workload should dictate the cluster design.
A configuration store for microservices may require different design considerations than a distributed lock service, a secrets store, or a Kubernetes control plane.
Cluster design tradeoffs include considerations such as:

• Availability
• Sata durability after member failure
• Performance/throughput
• Self-healing

Availability Copy Chapter

Instance availability on AWS is ultimately determined by the Amazon EC2 Region Service Level Agreement (SLA) which is the policy by which Amazon describes their precise definition of a regional outage.

In the context of an etcd cluster, this means that a cluster must contain a minimum of three members where EC2 instances are spread across at least two AZs in order for an etcd cluster to be considered highly available at a Regional level.

For most usecases, the additional latency that associated with a cluster spanning across Availability Zones, will introduce a negligible performance impact.

Availability considerations apply to all components of an application; if the application which is accessing the etcd cluster is only be deployed to a single Availability Zone, then it may not make sense to make the etcd cluster highly available across zones.

Data durability after member failure Copy Chapter

A highly available etcd cluster is resilient to member loss, however, it is important to consider data durability in the event of a disaster when designing an etcd deployment.
Deploying etcd on AWS supports multiple mechanisms for data durability. These mechanisms are:

• Replication: etcd replicates all data to all members of the etcd cluster.
  Therefore, given more members in the cluster and more independent failure domains, the more less-likely it is that the data stored in an etcd cluster will be permanently lost in the event of a disaster.

• Point in time etcd snapshotting: The etcd v3 API introduced support for snapshotting clusters.
  The operation is cheap enough (completing in the order of minutes) to run quite frequently and the resulting archives can be archived in a storage service like Amazon Simple Storage Service (S3).

• Amazon Elastic Block Storage (EBS): An EBS volume is a replicated, network attached block device which have stronger storage safety guarantees than InstanceStore, which in turn, has a life cycle associated with the life cycle of the attached EC2 instance.
  The life cycle of an EBS volume is not necessarily tied to an EC2 instance and can be detached and snapshotted independently which means that a single node etcd cluster backed by an EBS volume can provide a fairly reasonable level of data durability.

Performance/Throughput Copy Chapter

The performance of an etcd cluster is roughly quantifiable through latency and throughput metrics which are primarily affected by disk and network performance. Detailed performance planning information is provided in the performance section of the etcd operations guide.

Network Copy Chapter

AWS offers EC2 Placement Groups which allow the collocation of EC2 instances within a single Availability Zone which can be utilized in order to minimize network latency between etcd members in the cluster.
It is important to remember that collocation of etcd nodes within a single AZ will provide weaker fault tolerance than distributing members across multiple AZs.
Enhanced networking for EC2 instances may also improve network performance of individual EC2 instances.

Disk Copy Chapter

AWS provides two basic types of block storage: EBS volumes and EC2 Instance Store.
As mentioned, an EBS volume is a network attached block device while instance storage is directly attached to the hypervisor of the EC2 host.
EBS volumes will generally have higher latency, lower throughput, and greater performance variance than Instance Store volumes.
If performance, rather than data safety, is the primary concern, then it is highly recommended that instance storage on the EC2 instances be utilized.
Remember that the amount of available instance storage varies by EC2 instance types which may impose additional performance considerations.

Inconsistent EBS volume performance can introduce etcd cluster instability.
Provisioned IOPS can provide more consistent performance than general purpose SSD EBS volumes.
More information about EBS volume performance is available from AWS and also Datadog has shared their experience with getting optimal performance with AWS EBS Provisioned IOPS in their engineering blog.

Self-healing Copy Chapter

While using an ASG to scale the size of an etcd cluster is not recommended, an ASG can be used effectively to maintain the desired number of nodes in the event of node failure.
The maintenance of a stable number of etcd nodes will provide the etcd cluster with a measure of self-healing.

Next steps Copy Chapter

The operational life cycle of an etcd cluster can be greatly simplified through the use of the etcd-operator.
The open source etcd operator is a Kubernetes control plane operator which deploys and manages etcd clusters atop Kubernetes.
While still in its early stages the etcd-operator already offers periodic backups to S3, detection and replacement of failed nodes, and automated disaster recovery from backups in the event of permanent quorum loss.

File: content/en/blog/2020/jepsen-343-results.md Copy Chapter

title: Latest Jepsen Results against etcd 3.4.3 date: 2020-01-30 author: Xiang Li Copy Chapter

Jepsen tested and analyzed etcd 3.4.3, and had both good results and useful feedback to share with us.

A key part of etcd's design is strong consistency guarantees across the distributed key-value store. Kubernetes, Rook, OpenStack, and countless other critical software projects rely on etcd, in part, because of the etcd project's focus on reliability and correctness.

Over the years, the etcd team has put tremendous effort on building testing and chaos engineering frameworks. We feel that we've improved our consistency, and have asked Jepsen for validation.

Jepsen is the leading company testing open source distributed systems to check if they fulfill their consistency guarantees. They first tested etcd in version 0.4.1, and we have been using these findings ever since to improve our consistency. In this blog post, we share the overall positive results of the Jepsen analysis of etcd v3.4.3 as well as our plans to resolve the issues found in the analysis.

Funding for Jepsen's work was provided by the Cloud Native Computing Foundation, which hosts etcd.

What went well Copy Chapter

During the numerous failure cases created by Jepsen against an etcd cluster, it continued to perform as designed. From the analysis, the Jepsen author says:

etcd 3.4.3 lived up to its claims for key-value operations: we observed nothing but strict-serializable consistency for reads, writes, and even multi-key transactions, during process pauses, crashes, clock skew, network partitions, and membership changes. Strict serializable behavior was the default for key-value operations.

Watches appear correct, at least over single keys. So long as compaction does not destroy historical data while a watch isn't running, watches appear to deliver every update to a key in order.

Since Jepsen never "passes" software, but rather reports a lack of prohibitive issues, this is a great result from them. In general, it's much better than other distributed databases tested to date.

We believe two factors contribute to this positive result to the rigorous Jepsen analysis:

1. A simple core Copy Section

A simple solution to a problem space usually results in a robust system. As an example, etcd's Raft implementation is one of the simplest implementations in open source. It focuses on the core state machine of Raft and avoids wall-time and I/O handling in the core library. And, in its API, etcd supports multi-key transactions, but adopts a simple transactional model to keep the system easier to comprehend.

2. Rigorous testing practice Copy Section

Early in the development cycle of etcd, we set a goal to achieve 80% unit test line coverage and 90% unit test line coverage over core components like Raft and the MVCC storage engine. As the project progressed, we put effort into ensuring logical path coverage for core components by introducing fail points. To embrace the randomness of timing, OS scheduling and asynchronous network I/Os, we also developed continuous integration tests, which run 24x7, to try and produce issues on a running system in a way that unit tests simply cannot. These tests have increased the quality of etcd significantly.

Identified Issues Copy Chapter

Of course, no system undergoes Jepsen testing without them reporting some areas where it could be better. etcd had a few:

Lock Issues Copy Section

One lock implementation issue was found during the test where etcd failed to check the ownership of the lock before the pending lock API call returned.

In etcd, a lock acquirer is associated with a session; the acquirer holds the lock until the session expires. When first attempting to acquire the lock, it may be held by someone else. In that case, the etcd server puts the acquirer into a queue, where it must wait until other lock holders release the lock. The problem is that the acquirer's session might expire during this wait time. The consequence of this premature expiration is that the etcd server must check the existence of the session again before returning to the API call.

Documentation Issues Copy Section

The Jepsen analysis not only covers software correctness but also verifies the documentation claims. The Jepsen author found documentation issues and misleading use case examples of etcd. The previous etcd documentation on consistency and isolation is not clear and might cause confusion.

Our documentation describes the consistency model based on the Wikipedia definition, which is also the classification some etcd engineers learned in University. It also separates the isolation level from consistency level, as there is no general agreement on how the two should be defined together.

The Jepsen author suggests etcd documentation adopt the term "strict serializability" originally defined in the Linearizability paper published by Herlihy and Wing. After reading the documentation on other open source projects, as well as listening to feedback from our users, the etcd team agrees with the Jepsen author that the suggested model and term are clearer and easier to understand.

Next, the lock API use case example is misleading and might cause runtime issues.

Like any other distributed lock system, an etcd lock provides different guarantees than say a local thread level lock. Specifically, an etcd lock only safely guarantees mutual exclusion inside etcd's own keyspace and transactions with revision checking. It provides weaker guarantee when accessing external resources with dependency on timing. Deadlock prevention and lock invalidation is difficult with distributed locks because a distributed lock does not protect resources within the same process, or even the same machine. To address this, distributed locks traditionally rely on a lease and heartbeat mechanism to detect disconnected lock holders and invalidate locks. When a lock holder is disconnected or paused without using revision checking (a fencing token implementation in etcd), it might access the protected resources simultaneously with the new lock holder. More details can be found in Kleppmann's blog post.

The etcd lock documentation fails to point out this difference and provides inaccurate examples of using the lock. We agree with the Jepsen team that the issues must be resolved, and etcd users should be informed with the accurate claims.

Addressing Identified Issues Copy Chapter

All the issues mentioned above are being addressed by the etcd team and the Jepsen team. Here is the list of issues and fixes. We appreciate feedback from community over these issues and help the etcd project become better.

Future Copy Chapter

Jepsen analysis is not a one time effort. During the analysis, the Jepsen team set up an extensive testing framework for the etcd project specifically. It also enables etcd team and the community to run those tests anytime and catch problems in the future.

We would love to see someone in the etcd community integrate the etcd Jepsen tests directly into the existing etcd release pipeline. We hope to ensure that all future releases of etcd are Jepsen tests approved.

Besides Jepsen analysis, etcd community always welcome contributions related to correctness and reliability. We are excited about the results of this testing, and will remain vigilant while building a well engineered and correct product.

To learn more read the full Jepsen report for etcd 3.4.3.

File: content/en/blog/2021/announcing-etcd-3.5.md Copy Chapter

title: Announcing etcd 3.5 spelling: cSpell:ignore Gyuho author: "[Gyuho Lee](https://github.com/gyuho), Amazon Web Services" date: 2021-06-15 draft: false Copy Chapter

When we launched etcd 3.4 back in August 2019, our focus was on
storage backend improvements, non-voting member and pre-vote features. Since then,
etcd has become more widely used for various mission critical clustering and database
applications and as a result, its feature set has grown more broad and complex. Thus,
improving its stability and reliability has been top priority in recent
development.

Today, we are releasing etcd 3.5. The past two years
allowed for extensive iterations in fixing numerous bugs, identifying
optimization opportunities at scale, and evolving its surrounding ecosystem.
The etcd project also became a
CNCF graduated project
during this time frame. This release is the result of continuous evolution and
grungy, thankless tasks done by the etcd community.

In this blog post, we review the most notable changes to etcd 3.5, and present a
project road map for future releases. For a complete list of changes, see
CHANGELOG 3.5. For more updates, follow us on Twitter @etcdio. To get
etcd, see Install.

Security Copy Chapter

Given that etcd often handles sensitive data, improving and maintaining security
posture is our highest priority. In order to have a comprehensive understanding
of etcd security landscape, we completed third-party security audits: The first
report was published in February 2020, where we identified (and fixed) various
edge cases and high severity issues. For more information, see security audit
report.

To adhere to the highest levels of security best practices, etcd now has a
security release
process,
and runs automated tests with static analysis tools, such as errcheck,
ineffassign, and others.

Features Copy Chapter

The migration to structured logging is complete. etcd now defaults to
zap logger that has a reflection-free,
zero-allocation JSON encoder. We have deprecated
capnslog that logged with
reflection-based serialization.

etcd now supports built-in log rotation that configures rotate thresholds,
compression algorithms, etc. For more information, see hexfusion@ of Red Hat's code change.

etcd now emits more detailed tracing information for expensive requests, such as:

{
"caller":"traceutil/trace.go:116",
"msg":"trace[123] range",
"detail":"{
    range_begin:foo;
    range_end:fooo; response_count:100000; response_revision:191496;}",
"duration":"132.449773ms",
"start":"...:32.611-0700",
"end":"...:32.744-0700",
"steps":[
    "trace[123] step 'range keys from bolt db' (duration: 92.521911ms)",
    "trace[123] step 'filter and sort the key-value pairs' (duration: 22.789099ms)"]}

This provides a very useful signal for understanding the lifetime of a request
spanning multiple etcd server components. See YoyinZyc@'s code change (from
Google).

Each etcd cluster maintains its own cluster version, a value agreed on by the
quorum of the cluster. Previously, downgrading such cluster versions (e.g., etcd
minor version from 3.5 to 3.4) was not supported in order to protect against
incompatible changes.

Let's say we allow 3.3 node to join 3.4 cluster and send a
lease checkpoint request to the leader, which was introduced only in etcd 3.4.
When 3.3 node receives the lease checkpoint request, it fails to
handle the unknown apply request (see etcd server apply
code).
However, one might not use such a lease checkpointer feature and be willing to
risk incompatible changes in order to perform emergency rollbacks (e.g., major
defect in new etcd versions). To ensure such rollbacks are easy and reliable,
we've added a downgrade API that validates, enables, and cancels etcd version
downgrades. For more information, see YoyinZyc@ of
Google's code change .

By requiring quorum agreement, etcd cluster membership applies with the same level of consistency as writes.
Previously however, the member list call was directly
served from the server's local data, which could be stale. Now, etcd serves the
member list with a linearizable guarantee — if the server is disconnected from the
quorum, the member list call will fail. See jingyih@ of
Google's code change.

gRPC gateway endpoint is now stable under /v3/*. gRPC gateway generates an
HTTP API to make etcd gRPC-based HTTP/2 protocol accessible via HTTP/1, such as:

curl -X POST -L http://localhost:2379/v3/kv/put -d '{"key": "Zm9v", "value": "YmFy"}'

etcd client now uses the latest gRPC, v1.32.0, requires a new
import path "go.etcd.io/etcd/client/v3", and migrates balancer
implementation upstream. For more information, see ptabor@ of
Google's code change.

Bug fixes Copy Chapter

etcd reliability and correctness are of utmost importance. That is why we
backport all critical bug fixes to previous etcd releases. The following are
the notable bugs we identified and fixed during etcd 3.5 development:

1. Lease objects piling up caused memory leaks, and the solution was to clear the expired
   lease queue in the old leader. For details, see tangcong@ of Tencent's fix.
2. Ongoing compact operation caused deadlock in mvcc storage layer. For details, see tangcong@ of Tencent's fix.
3. etcd server restart had redundant backend database open operations and as a result, reloading
   40-million keys took over 5-minutes, and the solution reduced the restart time by half.
   For details, see tangcong@ of Tencent's fix.
4. If etcd crashed before completing defragmentation, the next defragment operation
   might have read the corrupted file. The solution was to ignore and overwrite the
   existing file. See jpbetz@ of Google's fix.
5. Client cancelling watch did not signal the server to create leaky watchers. The solution was to
   explicitly send a cancel request to the server. See jackkleeman@ of Apple's fix.

Performance Copy Chapter

Kubernetes, the most prominent user of etcd, queries the entire keyspace to list
and watch its cluster resources. This range query happens whenever the resource
is not found in the kube-apiserver reflector cache (e.g., requested etcd revision
has been compacted, see kube-apiserver v1.21
code),
often causing slowness in reads from overloaded etcd servers (see GitHub
issue). In such cases,
kube-apiserver tracing warns as below:

"List" url:/api/v1/pods,user-agent... (started: ...) (total time: 1.208s): Trace[...]: [1.208s] [1.204s] Writing http response done count:4346

And etcd warns as follows:

etcdserver: read-only range request key:"/registry/pods/" range_end:"/registry/pods0" revision:... range_response_count:500 size:291984 took too long (723.099118ms) to execute

Our deep-dive into etcd heap profile uncovered major inefficiency in the server
warning logger that had a redundant encoding operation only to compute the size
of range responses with
proto.Size
call. As a result, a large range query had up to 60% heap allocation overhead,
thus causing out-of-memory crashes, or OOM, in overloaded etcd servers (see
Figure 1). We optimized the protocol buffer message size operation
and as a result, reduced etcd memory consumption up to 50% during peak usage (see
Figure 2). It was a set of small code changes, but for years, such performance
achievements were invisible without extensive testing and workload simulation. See
chaochn47@ of Amazon Web
Services's investigation and patch to replace
proto.Size calls.

Figure 1: etcd heap usage during a slow request warn logging for computing response size of a protocol buffer message. 61% of the heap was allocated in the proto.Size call path that encodes all key-value pairs in the message to compute the size.

Figure 2: etcd heap usage during range query before and after replacing the proto.Size call. Optimizing the proto.Size call in the etcd server reduced memory usage up to 50%.

The etcd 3.4 release made backend read transactions fully concurrent by copying
transaction buffers rather than sharing between writes and concurrent reads (see
code change from 3.4 release).
However, such a buffering mechanism comes with unavoidable copy overhead and
negatively impacted write-heavy transaction performance, as creating concurrent
read transactions acquires a mutex lock which then blocks incoming write
transactions.

etcd 3.5 improvements further increase transaction concurrency.

1. If a transaction includes a PUT (update) operation, the transaction instead shares the transaction buffer between reads and writes (same behavior as 3.3) in order to avoid copying buffers. This transaction mode can be disabled via etcd --experimental-txn-mode-write-with-shared-buffer=false.

   The benchmark results show that the transaction throughput with a high write ratio has increased up to 2.7 times by avoiding copying buffers when creating a write transaction (see Figures 3 and 4).
   This benefits all kube-apiserver create and update calls that use etcd transactions (see etcd3 store v1.21 code).
   For more information, see wilsonwang371@ of ByteDance's code change and benchmark results.

   Figure 3: etcd transaction ratio with a high write ratio. The value at the top is the ratio of reads and writes. The first ratio, 0.125, is 1 read per 8 writes. The second ratio, 0.25, is 1 read for 4 writes. The value at the right bar represents the inverse ratio of transaction throughput before and after etcd/pull/12896. With the shared buffer approach for writes, the transaction throughput is increased up to 2.7 times.

   Figure 4: etcd transaction ratio with a high read ratio. The value at the top is the ratio of reads and writes. The first ratio, 4.0, is 4 reads per 1 write. The second ratio, 8.0, is 8 reads per 1 write. The value at the right bar represents the inverse ratio of transaction throughput before and after etcd/pull/12896. With the shared buffer approach for writes, the transaction throughput is increased up to 25%.

2. etcd now caches the transaction buffer to avoid the unnecessary copy operations. This speeds up concurrent read transaction creation and as a result, the transaction with a high read ratio has increased up to 2.4 times (see Figures 5 and 6). See wilsonwang371@ of ByteDance's code change and benchmark results.

   Figure 5: etcd transaction ratio with a high write ratio. The value at the top is the ratio of reads and writes. The first ratio, 0.125, is 1 read per 8 writes. The second ratio, 0.25, is 1 read per 4 writes. The value at the right bar represents the inverse ratio of transaction throughput before and after etcd/pull/12933. With the caching mechanism for read transactions, the transaction throughput is increased up to 1.4 times.

   Figure 6: etcd transaction ratio with a high read ratio. The value at the top is the ratio of reads and writes. The first ratio, 4.0, is 4 reads per 1 write. The second ratio, 8.0, is 8 reads per 1 write. The value at the right bar represents the inverse ratio of transaction throughput before and after etcd/pull/12933. With the caching mechanism for read transactions, the transaction throughput is increased up to 2.5 times.

Monitoring Copy Chapter

Long-running load tests revealed that etcd server misrepresented its real memory
usage by masking the impact of Go garbage collection. We discovered that etcd server with Go 1.12 changed the runtime to use MADV_FREE in Linux kernel, and as a result, reclaimed memory was not
reflected in the resident set size, or RSS, metric. This had made the etcd memory
usage metric inaccurately static, thus showing no sign of Go garbage collection.
To fix this monitoring problem, we compile etcd 3.5 with Go 1.16 that defaults
to MADV_DONTNEED on Linux. For more information, see Figure 7 and GitHub Go issue 42330.

Figure 7: etcd memory usage during range query before and after setting MADV_DONTNEED in Go runtime. When run with GODEBUG=madvdontneed=1" or Go v1.16 or later, the etcd server accurately reports its memory usage in the CloudWatch mem_used metric or other monitoring tools such as top.

Monitoring is a fundamental service for reliability and observability. Monitoring
enables individual service owners to understand the current state and identify
possible causes for problem reports. Known as telemetry, the goal is to detect early warning signs and
diagnose potential issues. etcd creates server logs with tracing
information and publishes Prometheus metrics.

This information helps us determine possible
service impacts and causes. However, when a request call chain spans
multiple external components (e.g., from kube-apiserver to etcd), it is harder
to identify the issue. In order to efficiently identify the root cause, we
added distributed tracing support using
OpenTelemetry: When the distributed tracing is
enabled, etcd now uses OpenTelemetry to produce a trace across the RPC call
chain and thus easily integrate with the surrounding ecosystem. See
Figure 8, lilic@ of Red Hat's proposal, and GitHub issue 12460.

Figure 8: Example jaeger tracing UI for etcd.

Testing Copy Chapter

Distributed systems are full of subtle edge cases. The fact that a bug may only
emerge under special circumstances warrants exhaustive test coverage beyond
simple unit testing. etcd runs integration, end-to-end, and failure injection
tests, which provide a reliable and faster way to validate each change. However,
as development continued with extended feature sets, flaky tests quickly piled
up draining our productivity. So, we took on the series of arduous tasks that
often required multiple hours of debugging to root cause of failures to
improve test quality. Some notable changes include the following:

• reduce unit tests runtime by half
• configure test logger
• simplify test data cleanup
• close gRPC servers after tests

Platforms Copy Chapter

etcd presubmit tests are now fast and reliable, but were mostly running on x86
processors. And there have been numerous requests to support other architectures
like ARM (see GitHub issue 12852) and
s390x (see GitHub issue 11163).
A self-hosted GitHub action runner
provides a consistent way of hosting various external test workers (see GitHub
issue 12856). Using the GitHub
action, etcd now runs tests on ARM-based AWS EC2 instances (Graviton),
thereby officially supporting ARM64 (aarch64) platform.
In addition, we introduced a mechanism to support other platforms and
categorized support tiers based on testing coverage. For more information,
see the Supported platforms documentation PR 273 and the Supported platforms documentation.

Developer experience Copy Chapter

To better support integration with external projects, etcd now fully
adopts the Go 1.16 module. This
brought challenges as the existing codebase made migration
arduous, causing concerns regarding adoption in the community). Tooling is an important
part of etcd development and as a result, we needed a better solution to support our contributor experience.

Using Go module enables clear
separation between server and client code, ease of change management for
dependency updates, and verifiable build system without a convoluted codebase for
vendoring. With reproducible builds in place, we eliminated the need for
vending dependency and as a result, we reduced the etcd codebase size by half.
See ptabor@ of Google's modularization proposal
and code change to remove the vendor
directory.

To better isolate dependency trees, the etcd command-line interface now has a new
administrative tool etcdutl (not etcdut-i-l), and the sub-commands
include etcdutl snapshot and etcdutl defrag: The etcdctl snapshot and etcdctl defrag commands in previous releases are deprecated.
This change aligns well with the new Go module layouts: etcdctl solely relies
on client v3 libraries, whereas etcdutl might depend on etcd server-side
packages, such as bolt and backend database code. See ptabor@ of Google's code change.

To support inclusive naming initiatives, etcd
projects renamed the default branch master to main (see etcd-io/etcd
change and etcd-io/website
change). The migration was
seamless as GitHub handles the necessary redirects (see GitHub
renaming).

Each etcd write incurs an append message in Raft thus fsynced to disk. However, such
persistence might not be desirable for testing. To work around this, we've added the
etcd --unsafe-no-fsync flag to bypass disk writes for Raft WAL entries. See
crawshaw@ of Tailscale's code change and GitHub
issue 11930.

Community Copy Chapter

The diversity of etcd end users keeps expanding: Cloudflare
relies on etcd for managing its data center,
Grafana Cortex stores its configuration data in etcd,
Netflix Titus uses etcd for managing its container workloads,
and Tailscale
runs its control plane on top of etcd.

We have also extended our team of vendor contributors.
In the etcd 3.5 release, we've added two core
maintainers; Wenjia Zhang
of Google, who's been leading etcd
community meetings and Kubernetes integration, and Piotr Tabor
of Google, who's been leading numerous bug
fixes and codebase modularization work.

The diversity of contributors is key to
building a sustainable, welcoming open source project and fostering manageable
work environments.

See the 2021 CNCF etcd project journey report for more information.

New etcd.io Copy Chapter

Since etcd joined Cloud Native Computing Foundation
(CNCF) in December 2018, we have
refactored all user-facing documentation into a dedicated repository
etcd-io/website and modernized its website
hosting with Hugo). The migration was a huge
undertaking that required multiple months of engineering efforts and
communication between maintainers. The very blog post you are reading now is
hosted on the new etcd.io, thanks to
lucperkins@,
chalin@ of CNCF,
nate-double-u@ of CNCF, and many other
community contributors.

Future roadmaps Copy Chapter

Traffic overloads can cause cascading node failures and as a result, scaling
such clusters becomes challenging and could impair the ability to recover from
quorum loss. With so many mission critical systems built on top of etcd,
defending etcd against overload is paramount. We will revisit the etcd
throttle feature to shed excessive loads gracefully. Currently, the etcd
project has two pending rate-limiter proposals: vivekpatani@ of Apple's proposal
and tangcong@ of Tencent's proposal.

Large range querying from kube-apiserver is still the most challenging source of
process crashing, as it is relatively unpredictable. Our heap profile on such
workloads found that the etcd range request handler decodes and holds the entire
response before sending it out to gRPC server, adding up to 37% heap
allocation. See Figure 9 and chaochn47@ of Amazon Web
Services's investigation.

Paginating range calls in client code doesn't fully address the issue,
because it entails additional consistency considerations and
still requires full relists for expired resources. For more
information, see kube-apiserver v1.21 code).
To work around this inefficiency, etcd needs to support range streams. We
will revisit yangxuanjia@ of JD's range stream proposal,
as it requires a significant level of effort to introduce such semantic changes
in etcd and in downstream projects.

Figure 9: etcd usage during a range query for listing Kubernetes pods. 37% of the heap was allocated in etcd mvcc rangeKeys to hold key-value pairs for creating a range query response.

In order to reduce maintenance overhead, we are completely deprecating
the etcd v2 API in favor of a more performant and widely adopted v3 API.
The v2 storage translation layer via etcd --experimental-enable-v2v3
remains experimental in 3.5 and to be removed in the
next release. For details, see
ptabor@ of Google's proposal.

Historically, etcd releases have been a large undertaking due to infrequency,
because of the large delta, and a need for release automation.
We will develop an automated release system that
is more accessible to the community.

File: content/en/blog/2022/etcd-integrates-continuous-fuzzing.md Copy Chapter

title: etcd Integrates Continuous Fuzzing spelling: cSpell:ignore Gyuho author: "[Adam Korczynski](https://twitter.com/AdamKorcz4), [David Korczynski](https://twitter.com/Davkorcz), [Sahdev Zala](https://twitter.com/sp_zala)" date: 2022-03-11 draft: false Copy Chapter

In the last few months, the team at Ada Logics has worked on integrating continuous fuzzing into the etcd project. This was an effort focused on improving the security posture of etcd and ensuring a continued good experience for etcds users. The fuzzing integration involved enrolling etcd in the OSS-Fuzz project and writing a set of fuzzers that would bring the test coverage of etcd up to a mature level. In total, 18 fuzzers were written, and eight bugs were found, demonstrating the work’s value for etcd both short term and long term. All fuzzers were implemented by way of go-fuzz and when running in OSS-Fuzz instrumented by way of libFuzzer, and as such, etcd uses state-of-the-art open source fuzzing capabilities.
The full report of the engagement can be found here.

The etcd project was created at CoreOS in 2013 and later joined the CNCF in 2018, from which it graduated in 2020. It is an open source, strongly consistent, distributed key-value store to reliably store data that a distributed system or cluster of machines needs to be accessed. It also provides highly desirable features like Watches to monitor changes to keys. etcd is a critical component of Kubernetes where it is used as the primary data store for cluster data such as the clusters state data and its data related to its desired state. Besides being a key component in Kubernetes, etcd is also used by many other distributed systems. Because of its wide usage, etcd is a critical part of the open source ecosystem to fuzz for reliability bugs and security vulnerabilities. The CNCF annual survey of 2021 found that 96% of companies are either using or evaluating Kubernetes, and etcd’s performance and security are important to continued business operations of these users.

What is fuzzing? Copy Chapter

Fuzzing is a technique used to automate parts of the software testing process by way of a form of stress testing. The key idea is to write a fuzzing harness similar to a unit—or integration—test that will execute the application under test with some arbitrary input. The fuzzing engine that will run the fuzzing harness then uses genetic algorithms to extrapolate inputs that will cause the code under test to execute uniquely, i.e., generate inputs that trigger new code execution paths. The goal is then to observe if the code under test misbehaves in the event of any of the generated inputs. Fuzzing has been effective in uncovering reliability bugs and vulnerabilities in software for more than two decades, and open source software is increasingly adopting the technique.

Etcd fuzzing overview Copy Chapter

In this engagement, the goal was to write a set of fuzzers that would cover a lot of the etcd codebase and integrate the setup into the open source fuzzing service OSS-Fuzz. OSS-Fuzz is a free service offered by Google for critical open source projects to run their fuzzers continuously and report any crashes. Continuous analysis is important due to fuzzing relying on genetic algorithms, which effectively means the fuzzers will improve over time, and OSS-Fuzz will run the fuzzers daily indefinitely. In addition to this, continuous analysis is crucial for capturing any regressions.

Etcd is written in the Go programming language, making it safe from memory-corruptions. Fuzzing Go will find panics such as slice/index out of range, nil-pointer dereferences, invalid type assertions, timeouts, out of memory. At the end of this engagement, eight issues were found, all of which were fixed. They are broken down as such:

At the end of this engagement, the fuzzers provide significant coverage of the etcd project, including critical parts such as the etcd server, WAL, the auth store, and the raft package.
During the engagement, Ada Logics found that only a few of the critical parts of etcd would be accessible with a byte slice or string but instead accepted complex types such as structs. An example of this is the fuzzers for the etcd server, which configures and sets up an etcd server and then creates a series of pseudo-randomized structs representing different requests sent to the server. To write these fuzzers, Ada Logics used go-fuzz-headers to deterministically create pseudo-random structs from the data provided by libFuzzer.

Closing thoughts Copy Chapter

The etcd team is thankful to CNCF and Chris Aniszczyk for providing the opportunity to work with Ada Logics to develop new fuzzers for etcd. The software security is taken seriously by the CNCF, and it earlier funded the etcd project for a third-party security audit. We also want to thank all the etcd maintainers and reviewers, especially Marek Siarkowicz, Piotr Tabor, and Benjamin Wang, for their quick reviews of the fixes. The fuzzing findings and fixes are valuable add-ons to the previous conclusions of the security audit. etcd project has efficient test suites, and code changes are backed by tests, but the newly developed fuzzers and findings have provided significant value to the project. During the fuzzing, only eight issues were found, which revalidated the high quality of the etcd code. The etcd team should maintain the newly developed fuzzers and build on them to continue code quality and security.

File: content/en/blog/2023/engage_with_etcd_at_kccncna2023.md Copy Chapter

title: Engage with the etcd project at KubeCon NA 2023 author: "[James Blair](https://github.com/jmhbnz), Red Hat" date: 2023-11-05 draft: false Copy Chapter

KubeCon NA 2023 in Chicago is just around the corner! This year, the etcd project has a diverse range of talks, tutorials, and even interactive contribfest sessions for you to get involved in . As a critical foundational pillar of the Kubernetes ecosystem, etcd's presence at Kubecon underscores its importance in ensuring all our Kubernetes clusters continue to have robust and reliable distributed persistent state.

Here's a detailed overview of what you can expect from the Etcd Project's presence at KubeCon NA 2023:

1. Schedule of talks Copy Chapter

Below is a list of talks on etcd, you can add these to your custom agenda by clicking the links for each talk through to https://kccncna2023.sched.com.

2. Contribfest - Learn from the maintainers and get involved Copy Chapter

Join the etcd contributor community at this hands on session. We'll be working on improving key features and testing for Etcd, and in the process we’ll teach those new to the project how to contribute. Etcd welcomes both new contributors and those who want to “level up”. Attendees should be familiar with programming in Go, using GitHub, and should bring a laptop on which they can do cloud-native development: either a Linux laptop or your own Github Devcontainer setup.

• Facillitators: Marek Siarkowicz & Wenjia Zhang, Google; James Blair & Josh Berkus, Red Hat
• Date & Time: Tuesday, November 7, 4:30pm CST

3. Project pavilion booth: Meet the etcd team Copy Chapter

Along with the engaging sessions above, the Etcd Project will also be staffing a booth in the project pavilion. This is a fantastic opportunity for you to interact directly with the etcd team, ask questions, and learn more about the project's latest developments. Whether you are a beginner exploring the world of Kubernetes or an experienced developer seeking insights into etcd's intricacies, the etcd project's booth is the place to stop by, say hello and ask any questions you might have.

4. SIG-etcd meet and greet Copy Chapter

Lastly, on Thursday, November 9 from 12:00pm - 3:00pm the SIG-etcd team will have a table at the Meet the Kubernetes Contributor Community event. If you have questions specifically about the new etcd kubernetes special interest group please stop by.

We hope to see you there Copy Chapter

Don't miss these opportunities to enhance your understanding of etcd and the pivotal role it plays in our Kubernetes ecosystem. Join us at KubeCon NA 2023 in Chicago, and let's explore the possibilities of etcd together!

Note: Schedule and session details are subject to change. Please refer to the official KubeCon NA 2023 schedule for the most up-to-date information.

File: content/en/blog/2023/how_to_debug_large_db_size_issue.md Copy Chapter

title: How to debug large db size issue? author: "[Benjamin Wang](https://github.com/ahrtr), VMware" date: 2023-01-04 draft: false Copy Chapter

Background

Users can configure the quota of the backend db size using flag --quota-backend-bytes. It's the max number of bytes
the etcd db file may consume, namely the ${etcd-data-dir}/member/snap/db file. Its default value is 2GB, and the
suggested max value is 8GB.

2GB is usually sufficient for most use cases. If you run out of the db quota, you will see error message etcdserver: mvcc: database space exceeded
when trying to write more data, and see alarm "NOSPACE" (see example below) when checking the endpoint status or health state. It would be better to figure out whether it's expected. It's exactly the reason why I provide this guide.

$ etcdctl endpoint status -w table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX |             ERRORS             |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+
| 127.0.0.1:2379 | 8e9e05c52164694d |   3.5.5 |   25 kB |      true |      false |         2 |          5 |                  5 |  memberID:10276657743932975437 |
|                |                  |         |         |           |            |           |            |                    |                 alarm:NOSPACE  |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+

$ etcdctl endpoint health -w table
+----------------+--------+------------+---------------------------+
|    ENDPOINT    | HEALTH |    TOOK    |           ERROR           |
+----------------+--------+------------+---------------------------+
| 127.0.0.1:2379 |  false | 1.850456ms | Active Alarm(s): NOSPACE  |
+----------------+--------+------------+---------------------------+

How to check db size

The easiest way is to execute ls -lrt ${etcd-data-dir}/member/snap command directly. For example, the db file is
5148672 bytes in the following example,

# ls -lrt /path-2-db-data-dir/member/snap/
total 5108
-rw-r--r-- 1 vcap vcap   13503 Nov 15 00:48 000000000000001f-0000000000c0433e.snap
-rw-r--r-- 1 vcap vcap   13503 Nov 15 12:33 000000000000001f-0000000000c1c9df.snap
-rw-r--r-- 1 vcap vcap   13502 Nov 16 00:17 000000000000001f-0000000000c35080.snap
-rw-r--r-- 1 vcap vcap   13503 Nov 16 12:03 000000000000001f-0000000000c4d721.snap
-rw-r--r-- 1 vcap vcap   13502 Nov 16 23:52 0000000000000023-0000000000c65dc2.snap
-rw------- 1 vcap vcap 5148672 Nov 17 02:44 db

The second way to check db size is to execute etcdctl endpoint status command. For example, it's about 5.1MB in the following example.
Note that if you do not set the --cluster flag, then you need to get all members' endpoints included in --endpoints, e.g,
--endpoints https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379.

# etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key --endpoints https://etcd-0:2379 endpoint status -w table --cluster
+------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|                 ENDPOINT                 |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://master-0.etcd.cfcr.internal:2379 | 17f206fd866fdab2 |   3.5.4 |  5.2 MB |     false |      false |        35 |   13024257 |           13024257 |        |
| https://master-2.etcd.cfcr.internal:2379 | 604ea1193b383592 |   3.5.4 |  5.1 MB |     false |      false |        35 |   13024257 |           13024257 |        |
| https://master-1.etcd.cfcr.internal:2379 | 9dccb73515ee278f |   3.5.4 |  5.1 MB |      true |      false |        35 |   13024257 |           13024257 |        |
+------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

You can also display the output using json format (See example below). The benefit is that you can get the both dbSize
and dbSizeInUse. Let's use the first endpoint as an example, its dbSize is 5177344 bytes, and dbSizeInUse is
2039808 bytes. It means that there are (5177344-2039808) bytes of free space, which can be reclaimed by executing the defragmentation operation.

# etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key --endpoints https://etcd-0:2379 endpoint status -w json --cluster
[
  {
    "Endpoint": "https://master-0.etcd.cfcr.internal:2379",
    "Status": {
      "header": {
        "cluster_id": 7895810959607866000,
        "member_id": 1725449293188291300,
        "revision": 9662972,
        "raft_term": 35
      },
      "version": "3.5.4",
      "dbSize": 5177344,
      "leader": 11370664597832739000,
      "raftIndex": 13024242,
      "raftTerm": 35,
      "raftAppliedIndex": 13024242,
      "dbSizeInUse": 2039808
    }
  },
  {
    "Endpoint": "https://master-2.etcd.cfcr.internal:2379",
    "Status": {
      "header": {
        "cluster_id": 7895810959607866000,
        "member_id": 6939661205564306000,
        "revision": 9662972,
        "raft_term": 35
      },
      "version": "3.5.4",
      "dbSize": 5148672,
      "leader": 11370664597832739000,
      "raftIndex": 13024242,
      "raftTerm": 35,
      "raftAppliedIndex": 13024242,
      "dbSizeInUse": 2031616
    }
  },
  {
    "Endpoint": "https://master-1.etcd.cfcr.internal:2379",
    "Status": {
      "header": {
        "cluster_id": 7895810959607866000,
        "member_id": 11370664597832739000,
        "revision": 9662972,
        "raft_term": 35
      },
      "version": "3.5.4",
      "dbSize": 5144576,
      "leader": 11370664597832739000,
      "raftIndex": 13024242,
      "raftTerm": 35,
      "raftAppliedIndex": 13024242,
      "dbSizeInUse": 2039808
    }
  }
]

etcd also exposes the following metrics (click here to see a full list of metrics),

Compaction & Defragmentation

etcd supports MVCC(Multi-Version Concurrent Control), and it keeps an exact history of its key spaces.
The compaction operation is the only way to purge history. But the free space will not be reclaimed automatically,
instead you should perform defragmentation to reclaim the free space after the compaction operation.
So usually Compaction + Defragmentation is the right way to purge history and reclaim unused storage space.

Please follow guide maintenance/#space-quota
to manually perform compaction and defragmentation operations.

Notes:

1. Compaction is a cluster-wide operation, so you only need to execute compaction once on whichever etcd member.
   Of course, it will not do any harm if executing it multiple times.
2. Defragmentation is a time-consuming task, so it's recommended to do it for each member one by one.
3. Defragmentation is an expensive operation, so you should do it as infrequent as possible. On the other hand,
   you also need to make sure any etcd member will not run out of the storage quota. etcdctl defrag [flags] can be used
   for defragmentation, but tools such as etcd-defrag can provide enhanced
   functionality and ease of use and should be considered to use.
4. There is a known issue that etcd might run into data inconsistency issue if it crashes in the middle of an online
   defragmentation operation using etcdctl or clientv3 API. All the existing v3.5 releases are affected, including 3.5.0 ~ 3.5.5.
   So please use etcdutl to offline perform defragmentation operation, but this requires taking each member offline one at a time.
   It means that you need to stop each etcd instance firstly, then perform defragmentation using etcdutl, start the instance at last.
   Please refer to the issue 1 in public statement.
5. Please run etcdctl alarm disarm if there is a NOSPACE alarm.

The following example shows you how to execute defragmention using etcdutl,

$ etcdutl defrag --data-dir  ~/tmp/etcd/infra1.etcd/

What data occupies most of the storage space

If the compaction + defragmentation doesn't help; in other words, if the db size is still exceeding
or close to the quota, then you need to figure out what data is consuming most of the storage space.

The straightforward way is to count the number of each kind of object. If the etcd cluster is supporting
Kubernetes apiserver, then execute command below to do the statistics.

In the following example, there are about 971K events, so obvious the events occupy most of the storage space.
The next step is to use your best judgement to figure out the root cause. Is it expected? Have you or your user
change the value of --event-ttl (apiserver flag, defaults to 1h)? Is it running into a known issue (e.g. 107170 )?

# etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key --endpoints https://etcd-0:2379 get /registry --prefix --keys-only | grep -v ^$ | awk -F '/'  '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr
971223 events
51011 pods
10008 replicasets
10008 deployments
3886 services
2555 secrets
2549 serviceaccounts
2514 configmaps
2507 namespaces
1947 endpointslices
506 leases
500 minions
500 csinodes
500 cns.vmware.com
77 clusterroles
64 clusterrolebindings
34 apiregistration.k8s.io
13 podsecuritypolicy
12 rolebindings
12 flowschemas
11 apiextensions.k8s.io
10 roles
8 prioritylevelconfigurations
3 masterleases
2 vmware.com
2 ranges
2 priorityclasses
2 daemonsets
2 controllerrevisions
1 validatingwebhookconfigurations
1 nsx.vmware.com
1 jobs
1 csidrivers

Usually when there are huge entries, it may take a long time to execute command above, so please set larger timeout values (see below), otherwise you may run into "context deadline exceeded".

--dial-timeout 10s --command-timeout 20s

If the etcd instance is not running, then you can use etcd-dump-db
to do the similar analysis(see example below). Note that you can build the binary etcd-dump-db using command go build with a golang version 1.19.x.

./etcd-dump-db iterate-bucket  /path-2-etcd-data-dir/member/snap/db  key --decode | egrep -o '"/registry.*' | cut -d'|' -f1 | grep -v ^$ | awk -F '/'  '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr
630 leases
150 masterleases
82 nsx.vmware.com
77 clusterroles
64 clusterrolebindings
54 secrets
48 serviceaccounts
33 apiregistration.k8s.io
18 pods
13 podsecuritypolicy
12 rolebindings
12 flowschemas
12 configmaps
10 services
10 roles
9 apiextensions.k8s.io
8 replicasets
8 prioritylevelconfigurations
8 minions
8 deployments
6 namespaces
5 endpointslices
3 csinodes
2 ranges
2 priorityclasses
2 daemonsets
2 controllerrevisions
1 validatingwebhookconfigurations
1 jobs

Solutions

If the behavior (db size exceeds the quota) is expected, then you can set a bigger value for --quota-backend-bytes.
You need to make sure your cloud provider supports this, otherwise the manual update might not survive across cluster
upgrading or VM recreating. Note that etcd (actually boltDB) maps the db file into memory directly, so a larger value
also means more memory usage. Just I mentioned in the beginning of this post, the suggested max value is 8GB. Of course,
If your VM has big memory (e.g. 64GB), it's OK to set a value > 8GB.

The other solution is to set per-resource etcd servers overrides using apiserver flag --etcd-servers-overrides.
In the following example, there are two etcd clusters; one for the normal usage, and the other dedicated to events.

--etcd-servers="https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379" \
--etcd-servers-overrides="/events#https://etcd-3:2379,https://etcd-4:2379,https://etcd-5:2379"

As a mitigation, you may also want to delete unneeded objects to free up some space. You will need to delete objects before running compaction and defragmentation to free up the space.

If the behavior (db size exceeds the quota) isn't expected, then you'd better figure out the root cause and resolve
it firstly. If you insist on applying solutions mentioned above, it can mitigate the issue instead of resolving it.

File: content/en/blog/2023/introducing-sig-etcd.md Copy Chapter

title: Introducing sig-etcd author: "[Han Kang](https://github.com/logicalhan), [Marek Siarkowicz](https://github.com/serathius), Frederico Muñoz" date: 2023-11-01 draft: false Copy Chapter

Special Interest Groups (SIGs) are a fundamental part of the Kubernetes project, with a substantial share of the community activity happening within them. When the need arises, new SIGs can be created, and that was precisely what happened recently.

SIG etcd is the most recent addition to the list of Kubernetes SIGs. In this article we will get to know it a bit better, understand its origins, scope, and plans.

The critical role of etcd Copy Chapter

If we look inside the control plane of a Kubernetes cluster, we will find etcd, a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data -- this description alone highlights the critical role that etcd plays, and the importance of it within the Kubernetes ecosystem.

This critical role makes the health of the etcd project and community an important consideration, and concerns about the state of the project in early 2022 did not go unnoticed. The changes in the maintainer team, amongst other factors, contributed to a situation that needed to be addressed.

Why a special interest group Copy Chapter

With the critical role of etcd in mind, it was proposed that the way forward would be to create a new special interest group. If etcd was already at the heart of Kubernetes, creating a dedicated SIG not only recognises that role, it would make etcd a first-class citizen of the Kubernetes community.

Establishing SIG etcd creates a dedicated space to make explicit the contract between etcd and Kubernetes api machinery and to prevent, on the etcd level, changes which violate this contract. Additionally, etcd will be able to adopt the processes that Kubernetes offers its SIGs (KEPs, PRR, phased feature gates, amongst others) in order to improve the consistency and reliability of the codebase. Being able to use these processes will be a substantial benefit to the etcd community.

As a SIG, etcd will also be able to draw contributor support from Kubernetes proper: active contributions to etcd from Kubernetes maintainers would decrease the likelihood of breaking Kubernetes changes, through the increased number of potential reviewers and the integration with existing testing framework. This will not only benefit Kubernetes, which will be able to better participate and shape the direction of etcd in terms of the critical role it plays, but also etcd as a whole.

About SIG etcd Copy Chapter

The recently created SIG is already working towards its goals, defined in its Charter and Vision. The purpose is clear: to ensure etcd is a reliable, simple, and scalable production-ready store for building cloud-native distributed systems and managing cloud-native infrastructure via orchestrators like Kubernetes.

The scope of SIG etcd is not exclusively about etcd as a Kubernetes component, it also covers etcd as a standard solution. Our goal is to make etcd the most reliable key-value storage to be used anywhere, unconstrained by any Kubernetes-specific limits and scaling to meet the requirements of many diverse use-cases.

We are confident that the creation of SIG etcd constitutes an important milestone in the lifecycle of the project, simultaneously improving etcd itself, and also the integration of etcd with Kubernetes. We invite everyone interested in etcd to visit our page, join us at our Slack channel, and get involved in this new stage of etcd's life.