kruise Documentation

Automated management of large-scale applications on Kubernetes (incubating project under CNCF)

5,235 stars Go Go 1 file ยท ~17,098 tokens #cloud-native#cloudnative#cncf#container#k8s#kruise#kubernetes#openkruise
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy kruise on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name kruise -p 3000:3000 --restart always openkruise/kruise:latest

1. Project Overview & Quickstart (openkruise/kruise)

File: README.md

Kruise








English | ็ฎ€ไฝ“ไธญๆ–‡

Introduction

OpenKruise (official site: https://openkruise.io) is a CNCF(Cloud Native Computing Foundation) incubating project.
It consists of several controllers which extend and complement the Kubernetes core controllers for workload and application management.

Key Features

Quick Start

You can view the full documentation from the OpenKruise website.

Get Your Own Demo with Alibaba Cloud

  • install Kruise on a Serverless K8S cluster in 3 minutes, try:

Users

Registration: Who is using Kruise

  • Alibaba Group, Ant Group, DouyuTV, Sto, Boss็›ด่˜
  • hangyinxiaofei, vanyitech, Dmall, Bringg, ไฝ็–†็ง‘ๆŠ€
  • Lyft, Ctrip, ไบซไฝๆ™บๆ…ง, VIPKID, zhangmen
  • xiaohongshu, bixin, ๆฐธ่พ‰็ง‘ๆŠ€ไธญๅฟƒ, ่ทŸ่ฐๅญฆ, ๅ“ˆๅ•ฐๅ‡บ่กŒ
  • Spectro Cloud, ihomefnt, Arkane Systems, Deepexi, ็ซ่Šฑๆ€็ปด
  • OPPO, Suning.cn, joyy, Mobvista, ๆทฑๅœณๅ‡คๅ‡ฐๆœจ็ฝ‘็ปœๆœ‰้™ๅ…ฌๅธ
  • xiaomi, Netease, MeiTuan Finance, Shopee, Esign
  • LinkedIn, ้›ช็ƒ, ๅ…ด็››ไผ˜้€‰, Wholee, LilithGames, Baidu
  • Bilibili, ๅ† ่ตขไบ’ๅจฑ, MeiTuan, ๅŒๅŸŽ

Contributing

You are warmly welcome to hack on Kruise. We have prepared a detailed guide CONTRIBUTING.md.

Community

Active communication channels:

  • Slack: OpenKruise channel (English)
  • DingTalk๏ผšSearch GroupID 23330762 (Chinese)
  • WeChat: Search User openkruise and let the robot invite you (Chinese)
  • Bi-weekly Community Meeting (APAC, Chinese):
    • Thursday 19:30 GMT+8 (Asia/Shanghai), Calendar
    • Join Meeting(DingTalk): Search GroupID 23330762 (Chinese)
    • Notes and agenda
  • Bi-weekly Community Meeting (English): TODO

Security

Please report vulnerabilities by email toย [email protected]. Also see ourย SECURITY.md file for details.

License

Kruise is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

File: docs/proposals/20201024-advancedcronjob.md

title: AdvancedCronJob Crd and Controller
authors:

  • "@rishi-anand"
    reviewers:
  • "@Fei-Guo"
  • "@FillZpp"
  • "@jzhoucliqr"
    creation-date: 2020-10-25
    last-updated: 2020-10-25
    status: implementable

Implementing AdvancedCronJob Crd and Controller

  • Implementing AdvancedCronJob to support Job/BroadcastJob or any other future CRD and to run it periodically at a given schedule.

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Summary

This controller will be very generic and will have implementations to help developers to run a job or any CRD in specific schedule.

Motivation

  • Developer may come across a use-case when some job needs to be executed on at a specific schedule.
  • Found same use-case requirement in Issues #251 and I got motivated to implement it

Goals

  • Implementing a custom controller for AdvancedCronJob which acts like CronJob but it schedules Job/BroadcastJob or other CRD

Proposal

  • Adding a new CRD and controller for AdvancedCronJob.
  • AdvancedCronJob should be able to reconcile Job/BroadcastJob or any other future CRD if required.
  • Once AdvancedCronJob is created, spec cannot be modified.
  • Adding webhook for validation of AdvancedCronJob.

User Stories

  • Implement a CRD which contains below fields and a controller which honors all the fields and reconciles accordingly.
text
/* Detailed source-code truncated for AI context efficiency. */

Story 1

Create a above CRD and implement the controller

Story 2

Add unit and integration test cases

Implementation History

  • 10/24/2020: Proposal discussion in an issue #215
  • 11/04/2020: Proposal submission

File: docs/proposals/20210305-sidecarset-hotupgrade.md

title: SidecarSet Hot Upgrade Feature
authors:

  • "@zmberg"
    reviewers:
  • "@Fei-Guo"
  • "@furykerry"
  • "@FillZpp"
    creation-date: 2021-03-05
    last-updated: 2021-03-05
    status: implementable

SidecarSet Hot Upgrade Feature

  • SidecarSet supports lossless hot upgrades for stateful sidecar containers, such as Istio Envoy

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Motivation

OpenKruise abstracts SidecarSet to manage the sidecar containers in Pod. Please refer to the details: OpenKruise SidecarSet.
User use SidecarSet to be easily injecting and upgrading sidecar containers, the current upgrade can only meet the requirements of stateless sidecar containers.
A stateless container can be upgraded directly through container restart, without migration of state between the old and new versions, for example: Logtail.

However, there are many scenarios for stateful sidecar containers, such as Istio Envoy, which cannot be upgraded by a direct restart.
The envoy intercepts all the traffic from the pod, and if you restart it directly, it will affect the availability of the business container.

In this proposal, we try to figure out a way to support lossless hot upgrades for stateful sidecar containers.

Proposal

The SidecarSet implements a hot upgrade mechanism of the stateful sidecar container by adding an empty container. As shown below:

yaml
apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: test-sidecarset
spec:
  containers:
  - name: sidecar-name
    ....
    # sidecar container upgrade method
    upgradeStrategy:
      upgradeType: HotUpgrade | ColdUpgrade
      hotUpgradeEmptyImage: {Empty Image}
  • upgradeType: HotUpgrade indicates hot upgrade for stateful sidecar container; ColdUpgrade is used for stateless sidecar container
  • hotUpgradeEmptyImage: when upgradeType=HotUpgrade, user needs to provide an empty container for hot upgrades

Note: hotUpgradeEmptyImage has the same configuration as the sidecar container, for example: command, lifecycle, probe, etc, but it doesn't do anything.

Hot upgrade consists of the following two processes:

  • inject hot upgrade sidecar containers
  • in-place hot upgrade sidecar container

Inject Containers

When the sidecar container upgradeStrategy=HotUpgrade, the SidecarSet Webhook will inject two containers into the Pod:

  1. {sidecarContainer.name}-1: as shown in the figure below: envoy-1, the container run the actual working sidecar container, such as envoy:1.16.0
  2. {sidecarContainer.name}-2: as shown in the figure below: envoy-2, the container run the hot upgrade empty container, and it doesn't have to deal with any real logic, as long as it stays in place, such as empty:1.0

Hot Upgrade

The SidecarSet Controller breaks down the hot upgrade process of the sidecar container into three steps:

  1. Upgrade: upgrade the empty container to the new version of the sidecar container, such as envoy-2.Image = envoy:1.17.0
  2. Migration: the process completes the state migration of stateful container, which needs to be done by the business itself according to its own container characteristics.
    PostStartHook completes the migration of the above process.
    (Note: PostStartHook must block during the migration, and exit when migration complete.)
  3. Reset: the step resets the old version sidecar container into empty container, such as envoy-1.Image = empty:1.0

The above is the complete hot upgrade process. If a Pod needs to be hot upgraded several times, the above three steps can be repeated.

Migration Mechanism

SidecarSet Hot upgrade provides an upgrade mechanism for stateful container. The specific state transfer process still needs to be implemented by the business itself.
State migration is a very important step in the hot upgrade process, so i will describe some of the key points below:

1. Coordination Mechanism

The state migration process starts with a coordination between the two sidecar containers to decide the migration direction (sidecar-1 -> sidecar-2 or sidecar-2 -> sidecar-1).
Since state migration is always from old -> new version. For this case OpenKruise injects the environment variable SIDECARSET_VERSION into the sidecar container by default.
The value of SIDECARSET_VERSION is the resourceVersion of the corresponding SidecarSet, so the larger one is the newer version compared with the two SIDECARSET_VERSION.
As shown below, envoy-2 is the new version of sidecar:

text
$ kubectl describe pods test-pod
  Name:         test-pod
  Namespace:    default
  Annotations:
                version.sidecarset.kruise.io/nginx-sidecar-1: 1097936677
                version.sidecarset.kruise.io/nginx-sidecar-2: 1098297016
  Status:       Running
  Containers:
    nginx-sidecar-1:
      Image:          nginx:1.18
      Environment:
        SIDECARSET_VERSION:   (v1:metadata.annotations['version.sidecarset.kruise.io/nginx-sidecar-1'])
    nginx-sidecar-2:
      Image:          nginx:1.19
      Environment:
        SIDECARSET_VERSION:   (v1:metadata.annotations['version.sidecarset.kruise.io/nginx-sidecar-2'])

Although the version of sidecar has been saved in the environment SIDECARSET_VERSION, it still needs a coordination between two sidecars to determine, and recommend to use shared Linux IPC, such as: TCP communication, shared memory, UDS, etc.
The specific business can decide by itself according to its own technology stack.

Future Planning

The current coordination mechanism also requires sidecar containers to implement intercommunication,
and this approach increases the development cost of sidecar containers. I will simplify this behavior in the next release by adding a SIDECARSET_VERSION_ALT environment variable. This is shown below:

text
$ kubectl describe pods test-pod
  Name:         test-pod
  Namespace:    default
  Annotations:
                version.sidecarset.kruise.io/nginx-sidecar-1: 1097936677
                version_alt.sidecarset.kruise.io/nginx-sidecar-1: 1098297016
                version.sidecarset.kruise.io/nginx-sidecar-2: 1098297016
                version_alt.sidecarset.kruise.io/nginx-sidecar-2: 1097936677
  Status:       Running
  Containers:
    nginx-sidecar-1:
      Image:          nginx:1.18
      Environment:
        SIDECARSET_VERSION:   (v1:metadata.annotations['version.sidecarset.kruise.io/nginx-sidecar-1'])
        SIDECARSET_VERSION_ALT: (v1:metadata.annotations['version_alt.sidecarset.kruise.io/nginx-sidecar-1'])
    nginx-sidecar-2:
      Image:          nginx:1.19
      Environment:
        SIDECARSET_VERSION:   (v1:metadata.annotations['version.sidecarset.kruise.io/nginx-sidecar-2'])
        SIDECARSET_VERSION_ALT: (v1:metadata.annotations['version_alt.sidecarset.kruise.io/nginx-sidecar-2'])

The environment variable SIDECARSET_VERSION_ALT records the version of another sidecar container, therefore,
no longer needs to communicate with each other, in three scenarios:

  1. SIDECARSET_VERSION>0, SIDECARSET_VERSION_ALT=0

This scenario indicates that the sidecar is not currently in the hot upgrade process and should perform an initialization operation (e.g., ListenFd)

  1. SIDECARSET_VERSION>0, SIDECARSET_VERSION_ALT>0, SIDECARSET_VERSION>SIDECARSET_VERSION_ALT

This scenario indicates that the sidecar is in the process of hot upgrade, and the current version of the sidecar is new, so state migration needs to be performed in the PostStartHook

  1. SIDECARSET_VERSION>0, SIDECARSET_VERSION_ALT>0, SIDECARSET_VERSION<SIDECARSET_VERSION_ALT

This scenario indicates that the sidecar is in the process of hot upgrade, and the current version of the sidecar is old, which will only happen if the old sidecar exits abnormally.
so it should not do any listening or other operations after starting, and just running there without doing anything.

2. State Migration

State migration is essentially the migration of stateful resources from the old version to the new version without interrupting service to the outside world. For example, for envoy, it is the migration of fd from the old sidecar to the new sidecar. fd can be passed between processes via Unix Domain Socket, but simply passing fd is still far from the goal of hot upgrade.
We can divide fd into two types: ListenFd and ConnectionFd, ListenFd is easier to migrate due to its stateless property, while ConnectionFd is not easy to complete the migration, the recommended migration solution is to migrate ListenFd, ConnectionFd "gracefully offline" and re-establish the connection to the new service process at the next request.

3. Exception Handling

  • new version of sidecar container keeps failing to start (non-postStartHook execution failure): Image, or some other reasons could cause the repeating start-up failure of the new version of sidecar container. In this failure, the status of the pod will be not ready, but the old version of sidecar still remains running.
    under the circumstances, SidecarSet upgrades may be blocked due to the MaxUnavailable restriction, unless a newer success version is released.
  • failure during state migration (e.g. ListenFd migration succeeded, ConnectionFd graceful offline failed): In this case, the state resources (e.g. ListenFd) in the old version of the sidecar container will not be automatically restored, and the kubelet will restart the sidecar container at a later time.
    In this scenario, PostStartHook may skip the state migration, and performs initialization operations to complete the start of the new version sidecar.
  • old version of sidecar container exit abnormally: First of all, kubelet will restart the exited container, and after restarting, the sidecar container will first determine whether itโ€™s of old version. If so, it should just keep running but do nothing actually (such as listening for connections)

Other Notes

  1. For a single hot upgrade sidecar in the sidecarSet, two sidecar containers will be injected (resources.requests will double), so we recommend leaving the resources.requests unset, or set to half of the original resources.requests
  2. For the in-place hot upgrade process, if there is no exception in the process, the kubelet will not report any Pod.Status, and the pod will remain ready in the process.
    That is, the hot upgrade process will not update the endpoint data, for the client, the hot upgrade pod is continuously available, thus achieving the effect of lossless upgrade.

User Cases

A Sample Case

  1. Create sidecarSet with hot upgrade sidecar container
yaml
# sidecarset.yaml
apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: test-sidecarset
spec:
  selector:
    matchLabels:
      app: main
  containers:
  - name: nginx-sidecar
    image: nginx:1.18
    lifecycle:
      postStart:
        exec:
          # If the environment variable SIDECARSET_VERSION=1, this is the first time sidecar container has been started, and it exit without doing anything
          # If the environment variable SIDECARSET_VERSION>1, indicates that this is a hot upgrade of sidecar container,
          # then the script needs to complete the migration in the hot upgrade
          command:
          - /bin/bash
          - -c
          - /usr/local/bin/nginx-agent migrate
    upgradeStrategy:
      upgradeType: HotUpgrade
      hotUpgradeEmptyImage: empty:1.0.0
  1. Create pod
yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: main # matches the SidecarSet's selector
  name: test-pod
spec:
  containers:
  - name: main
    image: busybox:latest
    command:
    - /bin/sh
    - -c
    - sleep 100000
  1. Get pod with two sidecar containers
bash
$ kubectl get pods |grep test-pod
test-pod                            3/3     Running   0          79m

$ kubectl describe pods test-pod
Name:         test-pod
Namespace:    default
Status:       Running
Containers:
  nginx-sidecar-1:
    Image:         nginx:1.18
    ...
  nginx-sidecar-2:
    Image:         empty:1.0.0
    ...
  main:
   Image:         busybox:latest
   ...
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
  1. Update Sidecar Container Image
text
/* Detailed source-code truncated for AI context efficiency. */

Practical sample usage in service mesh

AliYun ASM has realized lossless upgrade of Data Plane based on the hot upgrade capability of SidecarSet.

For details, please refer to: AliYun ASM (help.aliyun.com/document_detail/193804.html).

Implementation History

  • 09/03/2021: Proposal submission

File: docs/proposals/20210316-containerrecreaterequest.md

title: Proposal Template
authors:

  • "@FillZpp"
    reviewers:
  • "@Fei-Guo"
  • "@furykerry"
    creation-date: 2021-03-16
    last-updated: 2021-03-23
    status: implementable

ContainerRecreateRequest

  • Provide a way to let users restart/recreate one or more containers in a Pod.

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Motivation

Now that the only way to recreate a container in an active Pod is to update it in-place.
However, it also requires users to change the image defined in workloads (e.g. CloneSet/Advanced StatefulSet).

We hope to provide a way to let users recreate(restart) one or more containers in any active Pods.
It is useful for some user scenarios, such as:

  1. Restart the application in app container.
  2. Restart a sidecar container along with app container.

Proposal

Define a CRD named ContainerRecreateRequest (short name CRR):

yaml
apiVersion: apps.kruise.io/v1alpha1
kind: ContainerRecreateRequest
metadata:
  namespace: pod-namespace
  name: xxx
  labels:
    crr.apps.kruise.io/pod-name: pod-name               # injected by kruise webhook
    crr.apps.kruise.io/node-name: node-name-of-the-pod  # injected by kruise webhook
spec:
  podName: pod-name
  containers:
  - name: app
  - name: sidecar
  strategy:
    failurePolicy: Fail                 # or 'Ignore'. If 'Fail', the CRR will finish when one container failed to stop or recreate
    orderedRecreate: false              # 'true' indicates to recreate the next container only if the previous one has recreated completely
    terminationGracePeriodSeconds: 30   # the optional duration in seconds to wait the container terminating gracefully
    unreadyGracePeriodSeconds: 3        # it is the optional duration in seconds to mark Pod as not ready over this duration before executing preStop hook and stopping the container
  activeDeadlineSeconds: 300        # the CRR will be marked as Completed immediately if it has ran over deadline duration since created
  ttlSecondsAfterFinished: 1800     # ttl after this ContainerRecreateRequest has completed
  1. The containers will be recreated one by one.
    Typically, we will stop the next one once if the previous has stopped, unless the unreadyGracePeriodSeconds is set.
  2. The unreadyGracePeriodSeconds depends on a new feature-gate named KruisePodReadinessGate,
    which indicates to inject a readinessGate during each Pod creating.
    Otherwise unreadyGracePeriodSeconds can only work for those new Pods created by Kruise that have the readinessGate.
  3. If users set ttlSecondsAfterFinished, then CRR will automatically be deleted after completed over this time.
    Otherwise, users have to delete the CRR manually.

Status

Status of CRR looks like this:

yaml
status:
  completionTime: "2021-03-22T11:53:48Z"
  containerRecreateStates:
  - name: app
    phase: Succeeded
  - name: sidecar
    phase: Succeeded
  phase: Completed

The status.phase can be:

  • Pending: this CRR waits to recreate
  • Recreating: this CRR is recreating
  • Completed: this CRR has completed, and status.completionTime is the timestamp of completion

Note that status.phase=Completed does not mean all containers in CRR have recreated successfully.
Users should find the information in status.containerRecreateStates.

The status.containerRecreateStates[x].phase can be:

  • Pending: this container waits to recreate
  • Recreating: this container is recreating
  • Failed: this container has failed to recreate
  • Succeeded: this container has succeeded to recreate

When the CRR has completed, only the containers in Succeeded phase are successfully recreated.

User Stories

Story 1

To restart the service running in one container, user needs to restart the specific container.
It is too EXPENSIVE to recreate the whole Pod.

Related Issue.

Requirements

ContainerRecreateRequest relies on kruise-daemon to stop the container in Pod.

So if the KruiseDaemon feature-gate is closed, ContainerRecreateRequest will also be disabled.

Implementation Details/Notes/Constraints

When users create a CRR, webhook will inject the current containerID and restartCount into spec.containers[x].statusContext.
And, when kruise-daemon starts to execute, it will skip the container if its containerID is not equal to the one in statusContext or the restartCount has been bigger,
which means the container has already been recreated (maybe by in-place update).

For containers, kruise-daemon will stop the container with or without preStop hook,
then kubelet will create a new container and start again.

While some containers being recreating, the other containers in the same Pod will still be running.

Risks and Mitigations

  • Recreating a container will cause the application in it to start again.
  • It is recreate rather than restart, so the files written in the rootfs of the previous container will be lost.
    But the files written in volume will be mounted to the new container.
  • There is no problem if the recreation occurs with an in-place update in the same time:
    • If Kubelet has stopped or recreated the container because of in-place update, kruise-daemon will consider it already recreated.
    • If kruise-daemon stops the container, Kubelet will keep to in-place update the container to the new image.

Implementation History

  • 16/03/2021: Proposal submission

File: docs/proposals/20210325-imagepredownload-during-inplace-update.md

title: Proposal Template
authors:

  • "@FillZpp"
    reviewers:
  • "@Fei-Guo"
  • "@furykerry"
    creation-date: 2021-03-26
    last-updated: 2021-03-26
    status: implementable

Pre-download image during in-place update

If a workload is going to update Pods in-place,
the controller could pre-download the new image on the Nodes of all Pods in the workload.

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Motivation

If the image has been pre-downloaded on Nodes, we can significantly accelerate the speed of in-place update,
for Kubelet has not to pull the image again.

Proposal

Add a new feature-gate: PreDownloadImageForInPlaceUpdate.

If it is opened, workloads like CloneSet, Advanced StatefulSet or SidecarSet will create one or more ImagePullJobs when it starts a new in-place update.

yaml
apiVersion: apps.kruise.io/v1alpha1
kind: ImagePullJob
metadata:
  namespace: cloneset-namespace
  name: cloneset-name-{random-id}
spec:
  image: nginx:1.9.1
  parallelism: 1
  podSelector:
    matchLabels:
      {the selector of CloneSet}
    matchExpressions:
      - key: controller-revision-hash
        operator: NotIn
        values:
        - {the updateRevision of CloneSet}
  completionPolicy:
    type: Always
  pullPolicy:
    backoffLimit: 1
    timeoutSeconds: 300

Default values:

  • The requested parallelism parallelism defaults to 1.
  • The timeout for pulling image timeoutSeconds defaults to 300.

Also, users can configure them in their workloads:

yaml
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
  annotations:
    apps.kruise.io/image-predownload-parallelism: 10
    apps.kruise.io/image-predownload-timeout-seconds: 600

User Stories

Story 1

Image pulling is the most time-consuming stage in an in-place update progress.

If the new image has already been downloaded on the Node, it makes in-place update more efficient.

Requirements

It relies on ImagePullJob and NodeImage to pre-download images.

So if the KruiseDaemon feature-gate is closed, PreDownloadImageForInPlaceUpdate can neither be opened.

Implementation Details/Notes/Constraints

  1. User updates an image in CloneSet template.
  2. If the cloneset-controller finds it is an in-place update,
    the controller will create an ImagePullJob to pull the new image on the Nodes of its Pods.
  3. When sorting Pods to update, CloneSet controller will firstly sort them by the image has already downloaded on their Nodes.
    Since there is many sorting criteria of pods to update, the image-downloaded sorting will be the lowest priority.
  4. After all Pods updated or the template been changed, CloneSet controller will delete the previous ImagePullJob.

Since different workloads have different sorting logics to update Pods in-place, so imagepulljob-controller will get the workload
from the owner reference of ImagePullJob, and call the sorting function of the workload controller to sort the Pods before pulling
images on their Nodes.

Risks and Mitigations

  • The ImagePullJob may caused the image downloading parallelly on multiple Nodes. So we have to limit the parallelism number.

Implementation History

  • 26/03/2021: Proposal submission

File: docs/proposals/20210331-cascadingdeletionprotection.md

title: Proposal Template
authors:

  • "@FillZpp"
    reviewers:
  • "@Fei-Guo"
  • "@furykerry"
    creation-date: 2021-03-31
    last-updated: 2021-03-31
    status: implementable

Cascading deletion protection

Provide a safety policy to protect Kubernetes resources from the cascading deletion mechanism.

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Motivation

Currently, there are so many risks can be caused by cascading deletion mechanism:

  1. delete a CRD mistakenly, all CR disappeared
  2. delete a Workload mistakenly, all Pods belongs to it deleted
  3. delete all Workloads mistakenly in batches, all applications in the cluster unavailable
  4. delete a namespace, all resources in it deleted

Kruise should provide a safety policy which could help users protect Kubernetes resources and
applications' availability from the cascading deletion mechanism.

Proposal

API definition:

  • a new feature-gate named ResourcesDeletionProtection
  • a label named policy.kruise.io/delete-protection, the values can be:
    • Always: this object will always be forbidden to be deleted, unless the label is removed
    • Cascading: this object will be forbidden to be deleted, if it has active resources owned

If the feature-gate has enabled, the resources below with this label will be validated for deletion operation:

Kind Group Version Cascading judgement
Namespace core v1 whether there is active Pods in this namespace
CustomResourceDefinition apiextensions.k8s.io v1beta1, v1 whether there is existing CRs of this CRD
Deployment apps v1 whether the replicas is 0
StatefulSet apps v1 whether the replicas is 0
ReplicaSet apps v1 whether the replicas is 0
CloneSet apps.kruise.io v1alpha1 whether the replicas is 0
StatefulSet apps.kruise.io v1alpha1, v1beta1 whether the replicas is 0
UnitedDeployment apps.kruise.io v1alpha1 whether the replicas is 0

User Stories

  1. delete a CRD mistakenly, all CR disappeared
  2. delete a Workload mistakenly, all Pods belongs to it deleted
  3. delete all Workloads mistakenly in batches, all applications in the cluster unavailable
  4. delete a namespace, all resources in it deleted

Requirements

If users enable ResourcesDeletionProtection feature-gate when install or upgrade Kruise,
Kruise will require more authorities:

  1. Webhook for deletion operation of namespace, crd, deployment, statefulset, replicaset and workloads in Kruise.
  2. By default, clusterRole for reading all resource types, because CRD validation needs to list the CRs of this CRD.
    Optionally, users can use manager.role.groupsAuthorization helm parameter to only limit which groups can be listed by Kruise.

Implementation Details/Notes/Constraints

Intercept deletion operation of the resources with policy.kruise.io/delete-protection:

  1. For a Namespace, list the active Pods in it
  2. For a workload, just decide by its replicas
  3. For a CRD, list the CRs of it using unstructured client

Risks and Mitigations

If all kruise-manager Pods are crashed or in other abnormal states, the deletion webhook will fail for the resources above,
which means these resources can not be deleted temporarily.

Implementation History

  • 31/03/2021: Proposal submission

File: docs/proposals/20210614-podunavailablebudget.md

title: Pod Unavailable Budget
authors:

  • "@zmberg"
    reviewers:
  • "@Fei-Guo"
  • "@furykerry"
  • "@FillZpp"
    creation-date: 2021-06-14
    last-updated: 2021-06-16
    status: implementable

Pod Unavailable Budget Feature

  • In voluntary disruption scenarios, PodUnavailableBudget can achieve the effect of preventing application disruption or SLA degradation, which greatly improves the high availability of application services.

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Motivation

In kubernetes, powerful workloads (e.g., deployment, statefulset, etc.) are often used to define and manage applications.
It not only manages the number of pods, but also provides powerful rolling capabilities such as MaxUnavailable, Partition, etc.
However, workload can only help the application reach the desired number of pods as fast as possible, but does not guarantee that the number of available pods will remain at the desired value at any given moment.

Kubernetes offers Pod Disruption Budget to help you run highly available applications even when you introduce frequent voluntary disruptions.
PDB limits the number of Pods of a replicated application that are down simultaneously from voluntary disruptions. However, it can only constrain the voluntary disruption triggered by the Eviction API.
For example, when you run kubectl drain, the tool tries to evict all of the Pods on the Node you're taking out of service.

In the following voluntary disruption scenarios, there are still business disruption or SLA degradation situations:

  1. The application owner update deployment's pod template for general upgrading, while cluster administrator drain nodes to scale the cluster down(learn about Cluster Autoscaling).
  2. The middleware team is using SidecarSet to rolling upgrade the sidecar containers of the cluster, e.g. ServiceMesh envoy, while HPA triggers the scale-down of business applications.
  3. The application owner and middleware team release the same Pods at the same time based on OpenKruise cloneSet, sidecarSet in-place upgrades

Proposal

PodUnavailableBudget is an attempt for voluntary disruption, which is not only compatible with Kubernetes PDB protection for Eviction API, but also able to support the protection ability of above scenarios.

API Definition

text
/* Detailed source-code truncated for AI context efficiency. */

Implementation

This program customizes the PodUnavailableBudget (later referred to as PUB) CRD resource to describe the desired state of the application, and the working mechanism is shown below:

  • Many voluntary disruption scenarios can be summarized as modifications to Pod resources, which are divided into the following three categories.
    • Modification of Pod metadata and spec section
    • DeletionPod, i.e. DeletionTimestamp assignment
    • Eviction API
  • PUB validating is based on the kubernetes admission webhook mechanism, which decides whether to allow this operation to be executed based on the UnavailableAllowed in the PUB Status:
    • When UnavailableAllowed>0, UnavailableAllowed--, the request is allowed
    • Modification of Pod: insert pod.Name in Status.UnavailablePods(when the Pod upgrade is complete, remove it)
    • Eviction or Deletion Pod: insert pod.Name in Status.DisruptedPods(when the Pod deleted, remove it)
    • When UnavailableAllowed<=0, the request is not allowed, indicates that the modification of the pod will be rejected
  • Pub Controller dynamically calculates Pub.Status according to selected Pods
    • DesiredAvailable indicates that minimum desired number of available pods(according to MaxUnavailable or MinAvailable)
    • CurrentAvailable indicates that current number of ready pods(condition.Ready=true)
    • UnavailableAllowed = CurrentAvailable - DesiredAvailable

Pub selector cannot overlap

Pub.selector selects the pods that this pub needs to protect, and the pod cannot match multiple pubs. So when creating a pub webhook validate will check all other pubs to make sure there is no overlap, e.g.

  1. Pub1 selector matchLabels[app] = webserver, Pub2 selector matchLabels[app] = webserver, so selector is overlap and pub2 will not be allowed to create
  2. Pub1 selector matchLabels[app] = webserver, Pub2 selector matchLabels[app] = db, so selector isn't overlap and pub2 will be allowed to create
  3. Pub1 selector matchLabels[app] = webserver, Pub2 selector matchLabels[backend] = true, because they don't contain the same label key, so they cannot be judged as overlap and pub2 will be allowed to create

Note: if there is a pod that contains both of the above labels (e.g. labels[app]=webserver, labels[backend]=true), then the pod will match the above Pub1 and Pub2. This is unexpected(undefined) behavior.

Comparison with PDB

Kubernetes PodDisruptionBudget implements protection against Pod Eviction based on the EvictionREST interface,
while PodUnavailableBudget intercepts all pod modification requests through the admission webhook validating mechanism (Many voluntary disruption scenarios can be summarized as modifications to Pod resources),
and reject the request if the modification does not satisfy the desired state of the PUB.

Pub contains all the protection capabilities of kubernetes PDB, you can use both, or use pub independently to implement your application protection.

Operational Notes

  • PUB keeps at most 2000 total entries across status.disruptedPods and status.unavailablePods. When this limit is reached, new disruption requests are rejected until the controller cleans old entries.
  • The controller keeps deleted pods in status.disruptedPods for up to 20 seconds before it assumes the deletion never completed and removes the stale record.
  • The controller keeps in-place updated pods in status.unavailablePods for up to 10 seconds after the pod becomes available again. This delay exists to absorb informer cache latency and avoid releasing quota too early.
  • The controller and pod mutating webhook stamp protected pods with pub.kruise.io/related-pub=. The legacy kruise.io/related-pub annotation is still written and read for backward compatibility during the v1beta1 promotion.

Implementation History

  • 06/14/2021: Proposal submission

File: docs/proposals/20210624-cloneset-scaledown-topology-spread.md

title: Proposal Template
authors:

  • "@FillZpp"
    reviewers:
  • "@Fei-Guo"
  • "@furykerry"
    creation-date: 2021-06-24
    last-updated: 2021-06-24
    status: implementable

CloneSet scale-down supports topology spread

Motivation

Currently, the sequence of CloneSet scale-down is:

  1. Node unassigned < assigned
  2. PodPending < PodUnknown < PodRunning
  3. Not ready < ready
  4. Lower pod-deletion cost < higher pod-deletion-cost
  5. Been ready for empty time < less time < more time
  6. Pods with containers with higher restart counts < lower restart counts
  7. Empty creation time pods < newer pods < older pods

What's more, CloneSet should also choose pods to delete according to topology constraints.

User Stories

  1. Pods stacking on the same node should be deleted preferentially.
  2. Should delete Pods according to topology spread constraints.

Implementation Details/Notes/Constraints

Add a rank step between (4) and (5). We can now support two Ranker types:

Note that if there are Pod Topology Spread Constraints defined in CloneSet template,
controller will use SpreadConstraintsRanker to get ranks for pods, but it will still sort pods in the same topology by SameNodeRanker.
Otherwise, controller will only use SameNodeRanker to get ranks for pods.

SameNodeRanker

SameNodeRanker helps calculate pod ranks according to same node stacking.

  • Rank of those pods that are alone on their nodes is 0 by default.
  • For a node that has multiple pods stacking on it, SameNodeRanker will sort these pods by active and use their indexes as ranks.

For example:

text
node-a: pod-0, pod-1
node-b: pod-2
node-c: pod-3, pod-4, pod-5

The result will be:

  • Rank of pod-2 is definitely 0.
  • Sort pods for node-a, if they are [pod-0, pod-1] after sorting, their ranks will be {pod-0: 1, pod-1: 0}.
  • Sort pods for node-c, if they are [pod-5, pod-3, pod-4] after sorting, their ranks will be {pod-5: 2, pod-3: 1, pod-4: 0}.

SpreadConstraintsRanker

SpreadConstraintsRanker is a little more complicated than SameNodeRanker.

First, we should define the constraint rule that the ranker should care about:

golang
type PodSpreadConstraint struct {
	TopologyKey     string `json:"topologyKey"`
	LimitedValues []string `json:"limitedValues,omitempty"`
}
  • TopologyKey is the key of node labels. Nodes that have a label with this key and identical values are considered to be in the same topology.
  • LimitedValues defines the specific values of this topology key. By default, all values of this topology key are included.

Note that:

  1. The constraint rules will be got from CloneSet template (such as cloneset.spec.template.spec.topologySpreadConstraints),
    users have no need to configure them additionally.
  2. LimitedValues is currently not provided in Pod Topology Spread Constraints,
    reserve for future extensions

SpreadConstraintsRanker sorting logic:

  1. Split nodes and pods into different values of such topology key.
  2. For all pods in one topology value, SpreadConstraintsRanker firstly sorts them by active and SameNodeRanker,
    then calculates their ranks as their index of this topology value.

For example:

text
zone-x:
  node-a: a-0, a-1
  node-b: b-0
  node-c: c-0, c-1, c-2

zone-y:
  node-d: d-0, d-1
  node-e: e-0, e-1

Constraint topologyKey: topology.kubernetes.io/zone

After sorting in same topology value, the sequence of pods in zone-x and zone-y will be:

text
zone-x: c-2, a-1, c-1, a-0, b-0, c-0
zone-y: d-1, e-1, d-0, e-0

Note that pods in same zone are sorted by same node spread.

So the final pod sequence by topology spread rank will be:
d-1, c-2, e-1, a-1, c-1, d-0, a-0, b-0, e-0, c-0

File: docs/proposals/20210628-workloadspread.md

title: WorkloadSpread
authors:

  • "@BoltsLei"
    reviewers:
  • "@Fei-Guo"
  • "@furykerry"
  • "@FillZpp"
    creation-date: 2021-06-28
    last-updated: 2021-07-19
    status: implementable

WorkloadSpread

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Motivation

Define a CRD named WorkloadSpread, which can support configuration and management for elastic and multi-domain applications,
even more powerful and flexible than UnitedDeployment in some ways.

Not like the workload type, WorkloadSpread can be used with those stateless workloads, such as CloneSet, Deployment, ReplicaSet and even Job.

So users needn't learn the usage of a new workload and how to deploy and update their application by it.
They only have to create WorkloadSpread CRs related to their existing workloads. Everything is just great!

Proposal

WorkloadSpread definition looks like this:

yaml
apiVersion: apps.kruise.io/v1alpha1
kind: WorkloadSpread
metadata:
  namespace: ...
  name: ...
  # ...
spec:
  targetRef:
    apiVersion: apps/v1 | apps.kruise.io/v1alpha1
    kind: Deployment | CloneSet
    name: workload-xxx
  subsets:
  - name: subset-a
    requiredNodeSelectorTerm:
      matchExpressions:
      - key: topology.kubernetes.io/zone
        operator: In
        values:
        - zone-a
#   preferredNodeSelectorTerms:
#   - weight: 50
#     preference:
#     matchExpressions:
#     - key: topology.kubernetes.io/zone
#       operator: In
#       values:
#       - zone-a
    maxReplicas: 3
    tolertions: []
    patch:
      metadata:
        labels:
          xxx-specific-label: xxx
  - name: subset-b
    requiredNodeSelectorTerm:
      matchExpressions:
      - key: topology.kubernetes.io/zone
        operator: In
        values:
        - zone-b
  scheduleStrategy:
    type: Adaptive | Fixed
    adaptive:
      disableSimulationSchedule: false
      rescheduleCriticalSeconds: 30
status:
  observedGeneration: 12
  subsetStatuses:
  - name: subset-a
    missingReplicas: 2
    creatingPods:
      pod-xxx: "2021-04-27T08:34:11Z"
    deletingPods:
      pod-yyy: "2021-04-27T08:34:11Z"
    subsetUnscheduledStatus:
      unschedulable: false
      unscheduledTime: 2021-04-27T08:34:11Z
      failedCount: 1
  - name: subset-b
    missingReplicas: -1

spec:

  • targetRef: reference to a specific workload, it supports Deployment in apps group and CloneSet in apps.kruise.io group
  • subsets: list of subsets which defines different topologies
    • name: name of this subset
    • requiredNodeSelectorTerm: the additional node selector of this subset, it will be injected into each term of the pod.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms
    • preferredNodeSelectorTerms: the additional node selector of this subset, it will be appended into the pod.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution
    • tolerations: the additional tolerations of this subset, it will be injected into pod.spec.tolerations
    • patch: the specified strategic patch body of this subset, it will be patched to Pods that belong to this subset

      use runtime.RawExtension type

      yaml
      # patch metadata:
patch:
  metadata:
    labels:
      xxx-specific-label: xxx
      yaml
      # patch container resources:
patch:
  spec:
    containers:
    - name: main
      resources:
        limit:
          cpu: "2"
          memory: 800Mi
      yaml
      # patch container environments:
patch:
  spec:
    containers:
    - name: main
      env:
      - name: K8S_CONTAINER_NAME
        value: main
    • maxReplicas: the maximum number of Pods in this subset, it can be an absolute number or percentage string. maxReplicas can be null represents that there is no replicas limits in this subset.
  • scheduleStrategy: strategies for schedule
    • type: can be Fixed or Adaptive
      • Fixed means always choose the former subset for Pods unless the subset's maxReplicas has been satisfied
      • Adaptive means choose the right subset by sequence of subsets and also enough resources
    • adaptive:
      • disableSimulationSchedule: whether to disable the simulation schedule in webhook

        webhook can take a simple general predicates to check whether Pod can be scheduled into this subset,
        but it just considers the Node resource and cannot replace scheduler to do richer predicates practically.

      • rescheduleCriticalSeconds: the critical duration that controller will trigger a Pod to reschedule to another subset.

        rescheduleCriticalSeconds indicates how long controller will reschedule a schedule failed Pod to the subset that has
        redundant capacity after the subset where the Pod lives.
        If a Pod was scheduled failed and still in a pending status over rescheduleCriticalSeconds duration, the controller will reschedule it to the suitable subset.

status:

  • subsetStatuses: statuses of subsets defined in spec
    • missingReplicas: the missing number of this subset, most of the time it is (maxReplicas - the number of active Pods)

      missingReplicas > 0 indicates the subset is still missing missingReplicas pods to create.
      missingReplicas = 0 indicates the subset already has enough pods, and there is no need to create.
      missingReplicas = -1 indicates the subset's maxReplicas not set, then there is no limit for pods number.

    • creatingPods: the map of Pods that just allocated to this subset by webhook

      creatingPods contains information about pods whose creation was processed by
      the webhook handler but not yet been observed by the WorkloadSpread controller.
      A pod will be in this map from the time when the webhook handler processed the
      creation request to the time when the pod is seen by controller.
      The key in the map is the name of the pod and the value is the time when the webhook
      handler process the creation request. If the real creation didn't happen and a pod is
      still in this map, it will be removed from the list automatically by WorkloadSpread controller
      after some time.
      If everything goes smooth this map should be empty for the most of the time.
      Large number of entries in the map may indicate problems with pod creations.

    • deletingPods: the map of Pods been deleted or evicted that just noticed by webhook

      deletingPods is similar with creatingPods and it contains information about pod deletion.

    • subsetUnscheduledStatus: contains the details for the unscheduled subset.
      • unschedulable: unschedulable is true indicates this Subset cannot be scheduled. The default is false.

        Controller will set status to unschedulable if finding some unscheduled Pods belongs to this subset. However,
        it is temporary when unschedulable is true, and controller will recover subset to schedulable after 10 minutes.

      • unscheduledTime: last time for unscheduled.
      • failedCount: the number of subset was marked with unschedulable.

        failedCount just records the number of failures. Every failure can lead subset unschedulable for 10 minutes.

Requirements

Pod Webhook

WorkloadSpread requires Pod webhook to inject rules into Pods and notice the deletion and eviction of Pods.

So if the PodWebhook feature-gate is set to false, WorkloadSpread will also be disabled.

deletion-cost feature

CloneSet has supported deletion-cost feature in the latest versions.

Since Kubernetes 1.21, there is a new annotation definition on Pod:

yaml
controller.kubernetes.io/pod-deletion-cost: -100 / 100 / 0

(In 1.21, users need to enable PodDeletionCost feature-gate, and since 1.22 it will be enabled by default)

Implementation Details/Notes/Constraints

WorkloadSpread has both webhook and controller. Controller should collaborate with webhook to maintain WorkloadSpread's status together.

The webhook is responsible for injecting rules into pod, updating missingReplicas and recording the creation or deletion entry of Pod into map.

The controller is responsible for updating the missingReplicas along with other statics, and cleaning creatingPods and deletingPods map.

Replicas control

Pod creation

When a Pod is creating and its workload has a related WorkloadSpread, webhook will check the subsetStatuses in the WorkloadSpread one by one:

  1. If missingReplicas > 0 or missingReplicas = -1 and this subset is not unschedulable, choose this subset temporarily.
    (1). Update missingReplicas -= 1 and put this pod into creatingPods map and try to update subsetStatus.
    (2). If update to WorkloadSpread conflicts with other updates, get the WorkloadSpread again and go back to step (1).
    (3). if update to WorkloadSpread successfully, inject the rules of this subset into the Pod.
  2. If missingReplicas = 0, go to the next subset.
  3. If there is no available subset left, just let the Pod go.

Pod deletion

Also, when a Pod that belongs to a subset is being deleted or evicted, webhook will put it into the deletingPods map in subsetStatus and update missingReplicas += 1.

Update Pod Status

When a Pod that belongs to a subset changes status phase to 'succeed' or 'failed', which means Pod has been terminated lifecycle, webhook will update missingReplicas += 1.

Reschedule strategy

Reschedule strategy will delete unscheduled Pods that still in pending status. Some subsets have no sufficient resource can lead to some Pods unschedulable.
WorkloadSpread has multiple subset, so the unschedulable pods should be rescheduled to other subsets.

Controller will mark the subset containing unschedulable pods to unschedulable status and Webhook cannot inject Pod into this subset by check subset's status.
And then controller cleans up all unschedulable Pods to trigger workload creating new replicas, and webhook will skip unschedulable subset and chose suitable subset to inject.

The unscheduled subset can be kept for 10 minutes and then should be recovered schedulable to schedule Pod again by controller.

Deletion priority

We use deletion-cost feature to restrict replica numbers of each subset when scaling down.
If controller.kubernetes.io/pod-deletion-cost annotation is set, then the pod with the lower value will be deleted first.

We have four types for subset's Pod deletion-cost

  1. The number of active Pods in this subset <= maxReplicas, deletion-cost = 100. indicating the priority of Pods
    in this subset is higher than other Pods in workload.
  2. The subset's maxReplicas is nil, deletion-cost = 0.
  3. The number of active Pods in this subset > maxReplicas, two class: (a) deletion-cost = -100, (b) deletion-cost = +100.
    indicating we prefer deleting the Pod have -100 deletion-cost in this subset in order to control the instance of subset
    meeting up the desired maxReplicas number.
  4. The remaining pods do not belong to any subset, deletion-cost = -100.

If WorkloadSpread changes its subset maxReplicas, Pods will not be recreated, but WorkloadSpread will adjust deletion-cost annotation through the above algorithm.

To change scheduleStrategy of WorkloadSpread will keep the deletion-cost annotation of Pod.

Caution

When you adjust the subset's maxReplicas, you need to trigger the workload's rollout to make the existing Pods meeting the new topology spread.

The workload scales up when it's maxReplicas type is percent, but the new Pods don't meet the desired spread.
The possible reason is the race between the kruise informer and the controller-manager informer.
The controller doesn't update the latest workloadReplicas, leading to the corresponding missingReplicas can't be scaled up in time.

Alternative Considered

UnitedDeployment

Both UnitedDeployment and WorkloadSpread allow the workload to be distributed in different subsets, but the WorkloadSpread requires no workload api changes.
One can use vanilla CloneSet even Deployment while attaching extra topology constraints.

WorkloadSpread can also support the elastic scenario. For example, user has two topology subset: ECS and virtual-kubelet.
ECS should hold on 30 replicas, and the extra Pods should be scheduled to virtual-kubelet.

yaml
subsets:
    - name: ecs
      requiredNodeSelectorTerm:
        matchExpressions:
          - key: type
            operator: NotIn
            values:
              - virtual-kubelet
      maxReplicas: 30
    - name: vk
      requiredNodeSelectorTerm:
        matchExpressions:
          - key: type
            operator: In
            values:
              - virtual-kubelet
      tolerations:
        - key: virtual-kubelet.io/provider
          operator: Exists

If the workload instance is scaled out, but the WorkloadSpread CR is not adjusted accordingly, unexpected behaviors may happen.

NodeAffinity

Add nodeAffinity to the workload template, such as preferredDuringSchedulingIgnoredDuringExecution, which can be scheduled in multiple regions.
However, it cannot limit the replica numbers of a subset, and it is only effective when scaling out the workload, not effective when scaling in.

The internal implementation of WorkloadSpread is also based on nodeAffinity, but it will provide richer control for multiple subsets.

PodTopologySpread

You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains.
This can help to achieve high availability as well as efficient resource utilization.

  1. WorkloadSpread's subset maxReplicas can be a percentage string, which defines the percentage spread in multiple topology, but PodTopologySpread doesn't support this feature.
  2. For PodTopologySpread, there's no guarantee that the constraints remain satisfied when Pods are removed.
    For example, scaling down a Deployment may result in imbalanced Pods distribution.
    For WorkloadSpread, controller will guarantee the workload replicas meeting up maxReplicas by deletion-cost feature.

File: docs/proposals/20210713-sidecarsetimagepullsecrects.md

title: SidecarSet ImagePullSecrets

authors:

  • "@veophi"

reviewers:

  • "@Fei-Guo"
  • "@furykerry"
  • "@FillZpp"

creation-date: 2021-07-13 last-updated: 2021-07-13 status: implementable

SidecarSet ImagePullSecrets

  • Provide a way to pull images using Secret from private repositories for SidecarSet.

Table of Contents

A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting
any additional information provided beyond the standard proposal template.
Tools for generating a table of contents from markdown are available.

Motivation

In k8s, Secret is an easy and safe way to store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys.
One of the most common-used features of it is to pull images from private repositories. However, SidecarSet does not support this feature so far.

Proposal

Main idea: In this design, we separate the logic of Secret and SidecarSet.
In SidecarSet part, we only consider injecting their imagePullSecrets fields into Pod.
Users should manually distribute the required Secrets to all the namespaces that the SidecarSet may be instantiated.

API Definition

We add imagePullSecrets field in apis/apps/v1alpha1/sidecarset_types.go:

go
type SidecarSetSpec struct {
	...
	// List of the names of secrets required by pulling sidecar container images
	ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
}

User Story

A typical example:

yaml
# sidecarset.yaml
apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: my-sidecarset
spec:
  selector:
    ...
  updateStrategy:
    ...
  containers:
    - name: my-sidecar
      image: my-repository/centos:6.7
     ...
  imagePullSecrets:
    - name: my-secret

The image will be pulled from the private repository my-repository, if the secret my-secret has stored the correct username and password of this repository.
If the correct secret my-secret doesn't exist, it will fail to pull this image.

Implementation

We will merge the imagePullSecrets both in Pod and SidecarSet in pkg/webhook/pod/sidercarset.go file:

go
func (h *PodCreateHandler) sidecarsetMutatingPod(ctx context.Context, req admission.Request, pod *corev1.Pod) error {
	...
	//Inject imagePullSecrets
	pod.Spec.ImagePullSecrets = mergeSidecarSecrets(pod.Spec.ImagePullSecrets, sidecarSecrets)
	...
}

//Merge the secrets in both pod and sidecarset.
func mergeSidecarSecrets(secretsInPod, secretsInSidecar []corev1.LocalObjectReference) (allSecrets []corev1.LocalObjectReference) {
	secretFilter := make(map[string]bool)
	for _, podSecret := range secretsInPod {
		if _, ok := secretFilter[podSecret.Name]; !ok {
			secretFilter[podSecret.Name] = true
			allSecrets = append(allSecrets, podSecret)
		}
	}
	for _, sidecarSecret := range secretsInSidecar {
		if _, ok := secretFilter[sidecarSecret.Name]; !ok {
			secretFilter[sidecarSecret.Name] = true
			allSecrets = append(allSecrets, sidecarSecret)
		}
	}
	return allSecrets
}

Implementation History

  • 13/07/2021: Proposal submission

2. Official Technical Reference & Guides (openkruise/openkruise.github.io)

Website

This website is built using Docusaurus 3, a modern static website generator.

Installation

text
$ npm install

Local Development

text
$ npm run start

This command starts a local development server and opens up a browser window for default English site. Most changes are reflected live without having to restart the server.

text
$ npm run start -- --locale zh

This command starts a local development server and opens up a browser window for Chinese site. Most changes are reflected live without having to restart the server.

Build

text
$ npm run build

This command generates static content into the build directory and can be served using any static contents hosting service.

Deployment

text
$ GIT_USER=<Your GitHub username> USE_SSH=true yarn deploy

If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the gh-pages branch.

Document format checking

The file format check includes five types respectively:

  1. Check the number of titles in both English and Chinese
  2. Check the number of links in both English and Chinese
  3. Check the number of highlights in both English and Chinese
  4. Check struct examples in both Chinese and English
  5. English word spelling check

โ€ป: The language_tool_python module used in Check 5 detects word spelling errors.When it detects a problem with the word, it will go to the pre_dict.json file to determine, so just note some special words here.

Explore Related Technologies & Topics

View All 9,111+ Repositories โ†’

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#cloud-native Browse cloud-native catalog โ†’ #cloudnative Browse cloudnative catalog โ†’ #cncf Browse cncf catalog โ†’ #container Browse container catalog โ†’ #k8s Browse k8s catalog โ†’ #kruise Browse kruise catalog โ†’ โšก MCP Servers Model Context Protocol โ†’ ๐Ÿง  Agent Skills Prompt & rule files โ†’ ๐Ÿ”Œ OpenAPI Specs REST & Swagger schemas โ†’