python Documentation

Doc/README Copy Section

Building the documentation

Install the test requirements with:

$ pip install -r ../test-requirements.txt

Use make html to build the docs in html format.

Doc/Patch Types Copy Section

Introduction to Kubernetes Patch Types

In Kubernetes, patches are a way to make updates or changes to resources (like Pods, ConfigMaps, Deployments, etc.) without having to replace the entire resource. Patches allow you to modify specific parts of a resource while leaving the rest unchanged.

Types of Kubernetes Patches Copy Chapter

There are several types of patches that Kubernetes supports:

1. JSON Merge Patch (Standard JSON Patch)
2. Strategic Merge Patch
3. JSON Patch
4. Apply Patch (Server-Side Apply)

1. JSON Merge Patch Copy Chapter

• JSON Merge Patch is based on the concept of merging JSON objects. When you apply a patch, you only need to specify the changes you want to make. Kubernetes will take your partial update and merge it with the existing resource.

• This patch type is simple and works well when you need to update fields, such as changing a value or adding a new key.

Example Scenario: Copy Section

Imagine you have a Kubernetes Pod resource that looks like this:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: nginx
      image: nginx:1.14
    - name: redis
      image: redis:5

Now, you want to change the image of the nginx container from nginx:1.14 to nginx:1.16. Instead of sending the entire resource, you can send only the part you want to change, like this:

{
  "spec": {
    "containers": [
      {
        "name": "nginx",
        "image": "nginx:1.16"
      }
    ]
  }
}

When you send this patch to Kubernetes:

It will replace the image of the nginx container with the new one (nginx:1.16).
It will leave the redis container unchanged, because it's not included in the patch.

Example Code (Python): Copy Section

from kubernetes import client, config

def main():
    config.load_kube_config()
    v1 = client.CoreV1Api()

    namespace = "default"
    name = "mypod"

    patch = {
        "spec": {
            "containers": [
                {
                    "name": "nginx",
                    "image": "nginx:1.16"
                }
            ]
        }
    }

    v1.patch_namespaced_pod(name=name, namespace=namespace, body=patch, content_type="application/merge-patch+json")

if __name__ == "__main__":
    main()

After the JSON Merge patch Copy Section

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: nginx
      image: nginx:1.16  # Updated image version
    - name: redis
      image: redis:5  # Unchanged

2. Strategic Merge Patch Copy Chapter

Strategic Merge Patch is another type of patching mechanism, mostly used in Kubernetes, that allows updates to objects in a way that is more aware of the structure and semantics of the resource being modified. It is strategic because it understands the structure of the object, rather than blindly replacing it, and applies the changes in a smart way.

• The patch itself is typically a JSON or YAML object, which contains the fields to be updated
• Adds New Fields: You can use it to add new fields or modify existing ones without affecting the rest of the object.
• Handle Lists or Arrays: When dealing with lists (e.g., arrays or dictionaries), Strategic Merge Patch handles merging and updates intelligently.

Example of Strategic Merge Patch: Copy Section

let's suppose we have a yaml file Target Resource (Kubernetes ConfigMap):

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  key1: value1
  key2: value2
  list1:
    - item1
    - item2

Strategic Merge Patch

data:
  key1: updated_value1  # Update key1
  key3: value3          # Add new key3
  list1:
    - item1
    - item2
    - item3             # Add item3 to list1

Result after Strategic Merge Patch

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  key1: updated_value1  # key1 is updated
  key2: value2          # key2 is unchanged
  key3: value3          # key3 is added
  list1:
    - item1
    - item2
    - item3             # item3 is added to list1

3. JSON Patch Copy Chapter

• JSON Patch is a standard format that specifies a way to apply updates to a JSON document. Instead of sending a new or merged version of the object, JSON Patch describes how to modify the object step-by-step.
• Operation-Based: A JSON Patch is an array of operations that describe modifications to a target JSON object.
• Ideal when you need to perform multiple, specific operations on resource fields (e.g., replacing a value, adding new fields, or deleting specific values).

Patch Structure: Copy Section

A JSON Patch is an array of operations. Each operation is an object with:
• op: The operation type (e.g., add, remove, replace, etc.).
• path: A JSON Pointer string (defined in RFC 6901) that specifies the location in the document to apply the operation.
• value: (Optional) The new value to apply (used with operations like add, replace, or test).
• from: (Optional) Used in operations like move and copy to specify the source path.

Supported Operations for JSON Patch Copy Section

1. **add**

• Adds a value to a specified path.
• If the path already exists, it adds the value to a list or object.

Example:

{ "op": "add", "path": "/a/b/c", "value": "foo" }

2. **remove**

• Removes the value at the specified path.

Example:

{ "op": "remove", "path": "/a/b/c" }

3. **replace**

• Replaces the value at the specified path.
• Functionally similar to remove followed by add.

Example:

{ "op": "replace", "path": "/a/b/c", "value": "bar" }

4. **move**

• Moves a value from one path to another.

Example:

{ "op": "move", "from": "/a/b/c", "path": "/x/y/z" }

5. **copy**

• Copies a value from one path to another.

Example:

{ "op": "copy", "from": "/a/b/c", "path": "/x/y/z" }

6. **test**

• Tests whether a value at a specified path matches a given value.
• Used for validation in transactional updates.

Example:

{ "op": "test", "path": "/a/b/c", "value": "foo" }

Example: Applying a JSON Patch

Target JSON Document:

{
  "a": {
    "b": {
      "c": "foo"
    }
  },
  "x": {
    "y": "bar"
  }
}

JSON Patch:

[
  { "op": "replace", "path": "/a/b/c", "value": "baz" },
  { "op": "add", "path": "/a/d", "value": ["new", "value"] },
  { "op": "remove", "path": "/x/y" }
]

Result:

{
  "a": {
    "b": {
      "c": "baz"
    },
    "d": ["new", "value"]
  },
  "x": {}
}

4. Apply Patch (Server-Side Apply) Copy Chapter

Server-Side Apply is a feature in Kubernetes that allows you to declaratively update resources by specifying their desired state. It provides an intuitive and robust way to manage resources without having to manually modify every field. It tracks which fields belong to which manager, which helps prevent conflicts when multiple clients (such as different controllers or users) update the same resource.

Key Features:

• Declarative Management: You provide the desired final state, and Kubernetes ensures the actual state matches it.
• Conflict Detection: Ensures changes from different clients don’t conflict with each other.
• Field Ownership: Kubernetes tracks which client or manager owns which fields of a resource.

Example Scenario:

You have a ConfigMap and want to update certain keys but leave others unchanged.

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-config
  namespace: default
data:
  key1: value1
  key2: value2

Goal:

• You want to update key2 to new_value2 and
• add a new key key3 with a value value3.
• leave key1 unchanged

Apply Patch YAML(Desired State):

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-config
  namespace: default
data:
  key2: new_value2   # Update existing key
  key3: value3       # Add new key

Resulting ConfigMap (after apply):

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-config
  namespace: default
data:
  key1: value1          # Remains unchanged
  key2: new_value2      # Updated value
  key3: value3          # New key added

CONTRIBUTING Copy Section

Contributing Guidelines

How to become a contributor and submit your own code Copy Chapter

Contributor License Agreements Copy Section

We'd love to accept your patches! Before we can take them, we have to jump a couple of legal hurdles.

Please fill out either the individual or corporate Contributor License Agreement (CLA).

• If you are an individual writing original source code and you're sure you own the intellectual property, then you'll need to sign an individual CLA.
• If you work for a company that wants to allow you to contribute your work, then you'll need to sign a corporate CLA.

Follow either of the two links above to access the appropriate CLA and instructions for how to sign and return it. Once we receive it, we'll be able to accept your pull requests.

Composition of This Repository and Where/How to Contribute Copy Chapter

The Kubernetes Python client contains mostly files that are generated by the OpenAPI generator from this OpenAPI spec. In the repo there is also the utility part, which allows developers to create their own kubernetes clients (kubernetes/base). The base repo was once a submodule of the main repo, but is now integrated into the main repo. The archived code is available (here).

Where to Submit Your Patch Copy Section

The following folders are automatically generated. You will need to submit a patch to the upstream Kubernetes repo kubernetes or the OpenAPI generator repo openapi-generator.

• kubernetes/client
• kubernetes/test
• kubernetes/docs.

In this main repo, the following folders contain developer written codes and the patches should be submitted as pull requests here:

• kubernetes/base
• kubernetes/config
• kubernetes/dynamic
• kubernetes/e2e_test
• kubernetes/leaderelection
• kubernetes/stream
• kubernetes/utils
• kubernetes/watch
• examples
• scripts.

Contributing A Patch Copy Section

1. Submit an issue describing your proposed change to the repo in question.
2. The repo owners will respond to your issue promptly.
3. If your proposed change is accepted, and you haven't already done so, sign a Contributor License Agreement (see details above).
4. Fork the desired repo, develop and test your code changes. Add a test if possible.
5. Submit a pull request.

Adding Dependencies Copy Section

If your patch depends on new packages, add those packages to requirements.txt and/or setup.py. If these package are for testing only, add those to test-requirements.txt.

Commits Copy Section

Generally we would like to see one commit per pull request. However, if the pull request is reasonably large, the PR can be divided into several commits that make logical sense. The commit message should be clear and indicative of the aim of the fix. Sometimes multiple commits in a single pull request is acceptable if it meets the Kubernetes pull request guidelines.

If you have several commits in a pull request and have been asked to squash your commits, please use git reset --soft HEAD~N_COMMITS and commit again to make your PR a single commit.

Windows Developers Copy Section

The symbolic links contained in this repo do not work for Windows operating systems. If you are a Windows developer, please run the fix inside the scripts folder or manually copy the content of the kubernetes/base folder into the kubernetes folder.

Writing Tests Copy Section

In addition to running the fix yourself and telling us that your fix works, you can demonstrate that your fix really works by using unit tests and end to end tests. Tests are mainly located in three places. You should put your tests into the places that they fit in.

1. Generated tests by OpenAPI generator: these tests should pass and do not require modification.
2. End to end tests: these are tests that can only be verified with a live kubernetes server.
3. Base repo tests in the base repo, in which the test files are named test_*.py: These tests use the package Mock and confirms the functionality of the base repo files.

Coding Style Copy Section

We use an automatic coding style checker by using the diff of the autopep8 output and your code file. To make sure that your code passes the coding style checker, run autopep8 --in-place --aggressive --aggressive your_code.py before committing and submitting.

Running Tests Locally Copy Chapter

If you write a new end to end (e2e) test, or change behaviors that affect e2e tests, you should set up a local cluster and test them on your machine. The following steps will help you run the unit tests.

1. Acquire a local cluster. Minikube is a good choice for Windows and Linux developers. Alternatively if you are on Linux, you can clone the kubernetes repo and run install-etcd.sh and then local-up-cluster.sh to get a local cluster up and running.

2. Run the unit tests. In the root directory of the main repo, run python -m unittest discover.

3. Check the test results and make corresponding fixes.

CHANGELOG Copy Section

Breaking Change from upgrading OpenAPI Generator to v7.24.0

Both clients now use the modern Python generator instead of
python-legacy. The asynchronous client uses the generator's asyncio
library and aiohttp transport. Existing endpoint names, model names,
and wire aliases are preserved, but applications may need the following
updates:

• Synchronous runtime dependencies now require urllib3>=2.6.3,<3,
  pydantic>=2.11, lazy-imports>=1,<2,
  typing-extensions>=4.7.1, and python-dateutil>=2.8.2.
• Asynchronous runtime dependencies now require
  aiohttp>=3.13.5,<4.0.0, aiohttp-retry>=2.8.3, pydantic>=2.11,
  lazy-imports>=1,<2, typing-extensions>=4.7.1, and
  python-dateutil>=2.8.2. certifi and six are no longer direct
  asynchronous dependencies. Python 3.10 remains the minimum supported
  version for both clients.
• Models and API-call arguments now use Pydantic validation. Invalid,
  unknown, or previously coerced values can raise
  pydantic.ValidationError before a request is sent; models are
  keyword-only, reject unknown fields during construction, and validate
  assignments. The model local_vars_configuration argument and
  Configuration's discard_unknown_keys and
  disabled_client_side_validations arguments are removed.
  client_side_validation=False no longer disables generated
  validation.
• Low-level synchronous transport interfaces changed. Direct callers of
  ApiClient.request, ApiClient.call_api, ApiClient.deserialize, or
  the RESTClientObject HTTP-verb helpers must migrate to the modern
  request/response interface. ApiException.body is now decoded text
  instead of bytes.
• Low-level asynchronous transport interfaces also changed. Direct
  callers of ApiClient.call_api, ApiClient.param_serialize,
  ApiClient.response_deserialize, or RESTClientObject.request must
  migrate to the modern aiohttp request/response interface. HTTP
  sessions are created lazily; close an owned client with
  await client.close() or an async context manager. Interactive
  websocket streams use the generated _without_preload_content
  operations.
• Individual resource-deletion methods that previously returned
  V1Status, including CoreV1Api.delete_namespace and
  BatchV1Api.delete_namespaced_job, now return decoded dictionaries in
  both clients. A successful deletion can return either the deleted
  resource or a Status; access response fields with dictionary keys
  instead of model attributes.

See kubernetes-client/python#2631 and
kubernetes-client/gen#305.

v36.0.3

Kubernetes API Version: v1.36.2

Bug or Regression Copy Section

• Fix Watch.stream selecting watch instead of follow when streaming pod logs.
• Start the leader election worker thread as a daemon so it does not block process shutdown.
• Fix readline_channel, readline_stdout and readline_stderr crashing with OverflowError when using the default timeout, and returning None when a timeout expires.
• Fix format_quantity returning imprecise, non-canonical values for the milli, micro and nano suffixes, and ignoring quantize=Decimal(0).

v36.0.2

Kubernetes API Version: v1.36.1

Uncategorized Copy Section

• Restored backward compatibility for Configuration.auth_settings():
  the legacy api_key['authorization'] lookup is honored as a fallback
  when api_key['BearerToken'] is not set, fixing 401 Unauthorized
  regressions seen after upgrading to v36.0.0 (#2595). (#2604, @GK-07)

v36.0.1

Kubernetes API Version: v1.36.1

Bug or Regression Copy Section

• Fix load_incluster_config() and load_kube_config() (sync and async, with a static token) so requests carry an Authorization header on kubernetes-client/python v36+. Without this fix, in-cluster pods upgrading to v36 silently send unauthenticated requests and the apiserver rejects them as system:anonymous. (#2585, @Jmacek)

Deprecation Copy Section

• Support new exec v5 websocket subprotocol (#2486, @aojea)

v36.0.0

Kubernetes API Version: v1.36.1

v36.0.0b1

Kubernetes API Version: v1.36.1

Deprecation Copy Section

• Support new exec v5 websocket subprotocol (#2486, @aojea)

v36.0.0a3

Kubernetes API Version: v1.36.0

v36.0.0a2

Kubernetes API Version: v1.36.0

v36.0.0a1

Kubernetes API Version: v1.36.0

API Change Copy Section

• ACTION REQUIRED: DRA (Dynamic Resource Allocation) drivers and controllers now require granular RBAC permissions to update ResourceClaim statuses when the DRAResourceClaimGranularStatusAuthorization feature gate is enabled (beta in v1.36). Schedulers and controllers must be granted update/patch on resourceclaims/binding. DRA drivers must be granted associated-node:update or arbitrary-node:update (or patch equivalents) on resourceclaims/driver, restricted by their specific resourceNames. (kubernetes/kubernetes#134947, @aojea) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Scheduling and Testing]

• ACTION REQUIRED: Removed the integrated support for flex-volumes in kubeadm. Users were advised to migrate away from flex-volumes as recommended by SIG Storage since v1.22. If kubeadm users wish to continue using the feature, they need a custom image for the KCM that is not based on distroless, pass the KCM flag --flex-volume-plugin-dir, and mount the directory /usr/libexec/kubernetes/kubelet-plugins/volume/exec in the KCM static pod using kubeadm's extraVolumes mechanism before upgrading to v1.36. Previously, kubeadm automatically did the mounting if the user passed the flag. (kubernetes/kubernetes#136423, @neolit123) [SIG Cluster Lifecycle]

• ACTION REQUIRED: Renamed metric etcd_bookmark_counts to etcd_bookmark_total. If you are using custom monitoring dashboards or alerting rules based on the etcd_bookmark_counts metric, update them to use the new etcd_bookmark_total metric. (kubernetes/kubernetes#136483, @petern48) [SIG API Machinery, Etcd, Instrumentation and Testing]

• Added SchedulingConstraints to express topology-aware scheduling (TAS) constraints for PodGroup scheduling behind the TopologyAwareWorkloadScheduling feature gate. Added the TopologyPlacement plugin implementing the PlacementGenerate extension point to take constraints into consideration during PodGroup scheduling. (kubernetes/kubernetes#137271, @brejman) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Added DisruptionMode, PriorityClassName, and Priority fields to the Workload and PodGroup APIs to support workload-aware preemption when the WorkloadAwarePreemption feature gate is enabled. (kubernetes/kubernetes#136589, @tosi3k) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Added ImageVolumeWithDigest which includes the digest of image volumes in the container status. (kubernetes/kubernetes#132807, @iholder101) [SIG API Machinery, Apps, Node and Testing]

• Added MemoryReservationPolicy cgroup v2 MemoryQoS support to KubeletConfiguration for memory.min protection. (kubernetes/kubernetes#137584, @QiWang19) [SIG Node and Storage]

• Added spec.stubPKCS10Request to the Pod Certificates beta API to improve compatibility with existing certificate authority implementations that expect a PKCS#10 certificate signing request. spec.pkixPublicKey and spec.proofOfPossession were deprecated in favor of this field. (kubernetes/kubernetes#136729, @ahmedtd) [SIG API Machinery, Auth, Node and Testing]

• Added a deletion protection mechanism for PodGroup objects. (kubernetes/kubernetes#137641, @helayoty) [SIG API Machinery, Apps, Auth, Scheduling and Storage]

• Added alpha support (behind the PersistentVolumeClaimUnusedSinceTime feature gate) for tracking PersistentVolumeClaim unused status via a new Unused condition on PersistentVolumeClaimStatus. When enabled, the PVC protection controller sets Unused=True with a lastTransitionTime when no non-terminal Pods reference the PersistentVolumeClaim. (kubernetes/kubernetes#137862, @gnufied) [SIG Apps, Auth, Storage and Testing]

• Added alpha support for manifest-based admission control configuration (KEP-5793). When the ManifestBasedAdmissionControlConfig feature gate is enabled, admission webhooks and CEL-based policies can be loaded from static manifest files on disk via the staticManifestsDir field in AdmissionConfiguration. These policies are active from API server startup, survive etcd unavailability, and can protect API-based admission resources from modification. (kubernetes/kubernetes#137346, @aramase) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage, Testing and Windows]

• Added an admission plugin that validates PodGroup resources reference an existing Workload and match the declared PodGroupTemplate spec. (kubernetes/kubernetes#137464, @helayoty) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Added list-type support for attributes in DRA (KEP-5491). The DRAListTypeAttributes feature gate (disabled by default) activates the following enhancements:

  • DRA drivers can use list-type fields (bools/ints/strings/versions) for device attributes in ResourceSlice. The number of attribute values, including scalars and lists, per single device is limited to 48.
  • The matchAttribute/distinctAttribute constraints in ResourceClaim now work on both scalar and list attributes. The matchAttribute constraint matches when the intersection of all list values among candidate devices is non-empty. The distinctAttribute constraint (behind the ConsumableCapacity feature gate) matches when all list values among candidate devices are pairwise disjoint. Scalar values are implicitly treated as a singleton set.
  • Added a new CEL function .includes that works on both scalar and list attributes to test inclusion (e.g., device.attributes["dra.example.com"].model.includes("model-a")), supporting migration when a DRA driver changes an attribute value type from scalar to list or vice versa. (kubernetes/kubernetes#137190, @everpeace) [SIG API Machinery, Node, Scheduling and Testing]

• Added new concurrent-node-status-updates flag that is split from the concurrent-node-syncs flag. (kubernetes/kubernetes#136716, @yonizxz) [SIG Cloud Provider]

• Added opt-in alpha support in the kubeletplugin framework for DRA drivers to publish DRA Device metadata in Pod CDI mounts. (kubernetes/kubernetes#137086, @alaypatel07) [SIG Apps, Network, Node and Testing]

• Added opt-in scheduling behavior for CSI volumes. (kubernetes/kubernetes#137343, @gnufied) [SIG API Machinery, Scheduling and Storage]

• Added placement-based PodGroup scheduling algorithm to the scheduler. Its use is guarded by the TopologyAwareWorkloadScheduling feature gate. (kubernetes/kubernetes#136944, @brejman) [SIG Scheduling and Testing]

• Added stability-based lifecycle for declarative validation (Alpha/Beta/Stable). Scheduling Workload v1alpha1 now uses explicit declarative enforcement. (kubernetes/kubernetes#136793, @yongruilin) [SIG API Machinery and Scheduling]

• Added the PlacementGenerate extension point to the scheduler. It is used to generate placements for placement-based PodGroup scheduling. Its use is guarded by the TopologyAwareWorkloadScheduling feature gate. (kubernetes/kubernetes#137083, @brejman) [SIG Scheduling]

• Added the PlacementScore extension point to the scheduler for scoring placements in placement-based PodGroup scheduling, guarded by the TopologyAwareWorkloadScheduling feature gate. Deprecated MinNodeScore and MaxNodeScore in favor of MinScore and MaxScore. (kubernetes/kubernetes#137201, @brejman) [SIG Scheduling]

• Added the ResourcePoolStatusRequest API (v1alpha1) for querying DRA resource pool availability. External schedulers can discover available devices across pools before submitting workloads. Requires the DRAResourcePoolStatus feature gate (alpha). (kubernetes/kubernetes#137028, @nmn3m) [SIG API Machinery, Apps, Auth, Etcd, Instrumentation, Node, Scheduling, Storage and Testing]

• Added the --concurrent-resourceclaim-syncs flag to kube-controller-manager to configure ResourceClaim reconcile concurrency. (kubernetes/kubernetes#134701, @anson627) [SIG API Machinery, Apps, Node and Testing]

• Added the --tls-curve-preferences flag for configuring TLS key exchange mechanism. (kubernetes/kubernetes#137115, @damdo) [SIG API Machinery, Architecture, CLI, Cloud Provider, Node and Testing]

• Added the PodGroupPodsCount scheduler plugin to support workload-aware scheduling by prioritizing placements with higher Pod counts within a group. (kubernetes/kubernetes#137488, @vshkrabkov) [SIG Scheduling and Testing]

• Added the tlsServerName field to EgressSelectorConfiguration TLSConfig to allow overriding the server name used for TLS certificate verification. (kubernetes/kubernetes#136640, @kennangaibel) [SIG API Machinery, Apps, Auth, Storage and Testing]

• Added the alpha DRANodeAllocatableResources feature, which introduces a new ResourceSlice.Spec.Devices[*].NodeAllocatableResourceMappings field for DRA drivers to declare how device resources map to node allocatable Kubernetes resources (e.g., cpu, memory).(kubernetes/kubernetes#136725, @pravk03) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Added topology-aware scheduling (TAS) logic to the PodGroup scheduling cycle behind the TopologyAwareWorkloadScheduling feature gate, supporting scheduling of PodGroups on nodes with matching topology domains. (kubernetes/kubernetes#137489, @brejman) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Added validation to prevent negative duration values for imageMinimumGCAge. (kubernetes/kubernetes#135997, @ngopalak-redhat) [SIG API Machinery and Node]

• Changed deprecated sets.String with sets.Set[string] in apiserver admission subsystem. This is a breaking change for consumers of the NewLifecycle function. (kubernetes/kubernetes#134044, @mcallzbl) [SIG API Machinery and Auth]

• Clarified documentation and comments to indicate that the cpuCFSQuotaPeriod kubelet config field requires the CustomCPUCFSQuotaPeriod feature gate when using non-default values. No functional changes introduced. (kubernetes/kubernetes#133845, @rbiamru) [SIG Node and Release]

• Corrected OpenAPI schema union validation for the PodGroupPolicy struct in scheduling.k8s.io/v1alpha1. (kubernetes/kubernetes#136424, @JoelSpeed) [SIG API Machinery and Scheduling]

• DRA DeviceTaintRules: the TimeAdded field of the taint is now automatically updated when changing the effect. (kubernetes/kubernetes#137167, @pohly) [SIG API Machinery, Node and Testing]

• DRA: Added a spec.resourceClaims field to PodGroup resources for referencing ResourceClaims and ResourceClaimTemplates. Claims made by a PodGroup are reserved for the entire PodGroup instead of individual Pods, supporting more than 256 Pods sharing a single ResourceClaim. ResourceClaimTemplates referenced by a PodGroup's claim replicate into a ResourceClaim specific to that PodGroup, shared by all of the group's Pods. (kubernetes/kubernetes#136989, @nojnhuh) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• DRA: Graduated Device Binding Conditions (KEP #5007) to beta, enabled by default in v1.36. (kubernetes/kubernetes#137795, @ttsuuubasa) [SIG API Machinery, Node, Scheduling and Testing]

• DRA: Graduated device taints and tolerations (KEP #5055) to beta. Support for DeviceTaints in ResourceSlices is on by default. Support for DeviceTaintRules depends on enabling resource.k8s.io/v1beta2 and the DeviceTaintRules feature gate. (kubernetes/kubernetes#137170, @pohly) [SIG API Machinery, Apps, Auth, Cluster Lifecycle, Etcd, Node, Scheduling and Testing]

• Extended NodeResourcesFit to implement the PlacementScore extension point. The usage of the PlacementScore extension point is guarded by the TopologyAwareWorkloadScheduling feature gate. (kubernetes/kubernetes#136652, @brejman) [SIG Scheduling]

• Fixed fake.NewClientset() to work properly with correct schema. (kubernetes/kubernetes#131068, @soltysh) [SIG API Machinery]

• Fixed a few log calls that did not properly format their parameters. (kubernetes/kubernetes#137108, @pohly) [SIG API Machinery, Apps, Auth, Cluster Lifecycle, Network, Node, Scheduling and Testing]

• Fixed a potential nil pointer dereference in the scheduler's NodeResourcesFitArgs validation when using RequestedToCapacityRatio scoring strategy. (kubernetes/kubernetes#132120, @flpanbin) [SIG Scheduling]

• Fixed an issue in kube-apiserver, allowing it to recover from an established connection to an incorrect server that never returns the expected response during APIService availability checks. (kubernetes/kubernetes#137157, @bsalamat) [SIG API Machinery]

• For Pod resizes requested on nodes where the resize request exceeds the node's allocatable capacity or the node is running an OS that does not support resize, the request fails in admission rather than being marked as Infeasible in the Pod status later. (kubernetes/kubernetes#136043, @natasha41575) [SIG API Machinery, Node, Release, Scheduling, Storage and Testing]

• Generated fake.NewClientset which replaces the deprecated NewSimpleClientset for kube-aggregator and sample-apiserver. (kubernetes/kubernetes#136537, @soltysh) [SIG API Machinery]

• Graduated metric apiserver_storage_events_received_total to beta. (kubernetes/kubernetes#136314, @petern48) [SIG API Machinery, Etcd, Instrumentation and Testing]

• Graduated the ImageVolume feature to stable. (kubernetes/kubernetes#136711, @saschagrunert) [SIG Apps, Architecture, Node and Testing]

• Graduated the InPlacePodLevelResourcesVerticalScaling feature gate to beta, enabled by default. Pod-level CPU and memory resources can be resized in place for Pods with pod-level resources configured. (kubernetes/kubernetes#137684, @ndixita) [SIG API Machinery, Apps, Autoscaling, Node, Release, Scheduling and Testing]

• Graduated the UserNamespacesSupport feature gate to GA. (kubernetes/kubernetes#136792, @rata) [SIG API Machinery, Apps, CLI, Node, Storage and Testing]

• Graduated the config.k8s.io/flagz API to v1beta1. (kubernetes/kubernetes#137174, @richabanker) [SIG API Machinery, Instrumentation, Node, Scheduling and Testing]

• Graduated the config.k8s.io/statusz API to v1beta1. (kubernetes/kubernetes#137173, @richabanker) [SIG API Machinery, Instrumentation, Scheduling and Testing]

• HPA: Improved scaling to and from zero when the HPAScaleToZero feature gate is enabled. (kubernetes/kubernetes#135118, @johanneswuerbach) [SIG Apps, Autoscaling and Testing]

• Integrated Workload and PodGroup APIs with the Job controllers to support gang-scheduling. (kubernetes/kubernetes#137032, @helayoty) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Instrumentation, Node, Scheduling and Testing]

• Introduced scheduling.k8s.io/v1alpha2 Workload and PodGroup API to express workload-level scheduling requirements and let kube-scheduler act on those. Removed scheduling.k8s.io/v1alpha1 Workload API. (kubernetes/kubernetes#136976, @tosi3k) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling, Storage and Testing]

• Kube-apiserver: The --audit-policy-file config file now supports specifying group: "*" in resource rules to match all API groups. (kubernetes/kubernetes#135262, @cmuuss) [SIG API Machinery, Auth and Testing]

• Kube-controller-manager: Added ALPHA gauge metric informer_queued_items for informer queue length, published as informer_queued_items{name=kube-controller-manager,group=<group>,resource=<resource>,version=<version>} <count>. (kubernetes/kubernetes#135782, @richabanker) [SIG API Machinery, Architecture, Instrumentation and Testing]

• Kubelet: Added tiered cgroup v2 memory protection for MemoryQoS: memory.min for Guaranteed pods and memory.low for Burstable pods, with node-level metrics and rollback reconciliation (KEP-2570). (kubernetes/kubernetes#137719, @sohankunkerkar) [SIG Node, Storage and Testing]

• Locked the VolumeAttributesClass feature gate to true and updated the preferred storage version to storage.k8s.io/v1. (kubernetes/kubernetes#134556, @carlory) [SIG API Machinery, Apps, Etcd, Network, Node, Scheduling, Storage and Testing]

• Marked the endpoints field as optional in the OpenAPI spec for discovery.k8s.io/v1 EndpointSlice. This matches server behavior and resolves validation issues. (kubernetes/kubernetes#136111, @aojea) [SIG Network]

• Promoted DRAPrioritizedList to GA. (kubernetes/kubernetes#136924, @troychiu) [SIG Apps, Architecture, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Network, Node, Release, Scheduling, Storage and Testing]

• Promoted NodeDeclaredFeatures to beta. (kubernetes/kubernetes#136042, @pravk03) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation, Node, Scheduling, Storage and Testing]

• Promoted SnapshotMetadataService to v1beta1. Removed support for the v1alpha1 version. (kubernetes/kubernetes#137564, @iPraveenParihar) [SIG Storage and Testing]

• Promoted mutable CSI node allocatable count to GA. The MutableCSINodeAllocatableCount feature gate is locked to enabled. (kubernetes/kubernetes#136230, @torredil) [SIG API Machinery and Storage]

• Promoted several EndpointSlice metrics from alpha to beta stability. (kubernetes/kubernetes#136368, @bhope) [SIG Instrumentation and Network]

• Promoted several component-base metrics (kubernetes_build_info, rest_client_requests_total, rest_client_request_duration_seconds, running_managed_controllers) from Alpha to Beta stability, providing stronger API and label stability guarantees for consumers. (kubernetes/kubernetes#136154, @bhope) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scalability, Scheduling, Storage and Testing]

• Promoted several scheduler metrics (scheduler_goroutines, scheduler_permit_wait_duration_seconds, scheduler_plugin_evaluation_total, scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds, scheduler_unschedulable_pods) from alpha to beta stability, providing stronger API and label stability guarantees for metric consumers. (kubernetes/kubernetes#136155, @bhope) [SIG Instrumentation and Scheduling]

• Promoted the DRA extended resource feature to beta in v1.36. (kubernetes/kubernetes#135048, @yliaog) [SIG API Machinery, Architecture, Auth, Network, Node, Scheduling and Testing]

• Promoted the ConstrainedImpersonation feature to beta, enabled by default. (kubernetes/kubernetes#137609, @enj) [SIG API Machinery and Testing]

• Promoted the DRAAdminAccess feature gate to GA. (kubernetes/kubernetes#137373, @ritazh) [SIG API Machinery, Auth, Node, Scheduling and Testing]

• Promoted the MutatingAdmissionPolicy to GA (v1) in Kubernetes v1.36. The feature is now enabled by default. (kubernetes/kubernetes#136039, @lalitc375) [SIG API Machinery, Architecture, Etcd and Testing]

• Promoted the NodeLogQuery feature gate to GA. (kubernetes/kubernetes#137544, @jrvaldes) [SIG Node and Windows]

• Promoted the ProcMountType feature to GA. (kubernetes/kubernetes#137454, @haircommander) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing]

• Promoted the watch_list_duration_seconds metric from ALPHA to BETA. (kubernetes/kubernetes#136086, @richabanker) [SIG API Machinery, Instrumentation, Node and Testing]

• Promoted two Job controller metrics from alpha to beta stability, providing stronger API and label stability guarantees for metric consumers. (kubernetes/kubernetes#136367, @bhope) [SIG Apps and Instrumentation]

• Promoted workqueue metrics from ALPHA to BETA. (kubernetes/kubernetes#135522, @petern48) [SIG Architecture, Instrumentation and Testing]

• Removed CustomResourceDefinition stored versions from status upon StorageVersionMigrator migration. (kubernetes/kubernetes#135297, @michaelasp) [SIG API Machinery, Apps, Auth and Testing]

• Removed the in-tree Portworx volume plugin, completing the migration to CSI.
  Removed the GA CSIMigrationPortworx feature gate (locked since v1.33) and alpha InTreePluginPortworxUnregister feature gate, with all operations now redirected to CSI. (kubernetes/kubernetes#135322, @carlory) [SIG API Machinery, Apps, Auth, Node, Scalability, Scheduling, Storage and Testing]

• Removed the temporary build-tagged ProtoMessage() marker method implementations from Kubernetes REST API types in k8s.io/api, which had incorrectly identified them as standard v1 proto messages. Protobuf serialization of Kubernetes API types should use k8s.io/apimachinery/pkg/runtime/serializer/protobuf. (kubernetes/kubernetes#137084, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, Node, Scheduling and Storage]

• Slow requests that use impersonation can be tracked via the apiserver.latency.k8s.io/impersonation audit event annotation when the ConstrainedImpersonation feature is enabled. (kubernetes/kubernetes#137523, @enj) [SIG API Machinery, Auth and Testing]

• The DRAConsumableCapacity feature gate is enabled by default. (kubernetes/kubernetes#136611, @sunya-ch) [SIG API Machinery, Cluster Lifecycle, Node, Scheduling and Testing]

• The StrictIPCIDRValidation feature gate in kube-apiserver is enabled by default, meaning that API fields no longer allow IP or CIDR values with extraneous leading "0"s (e.g., 010.000.000.005 rather than 10.0.0.5) or CIDR subnet/mask values with ambiguous semantics (e.g., 192.168.0.5/24 rather than 192.168.0.0/24 or 192.168.0.5/32). (kubernetes/kubernetes#137053, @danwinship) [SIG Network and Testing]

• The kube-scheduler now updates PodGroup status with a PodGroupScheduled condition reflecting whether the group was successfully scheduled or is unschedulable. (kubernetes/kubernetes#137611, @helayoty) [SIG API Machinery, Apps, Scheduling and Testing]

• Updated API comments to reflect the stable state of Dynamic Resource Allocation (DRA). (kubernetes/kubernetes#136441, @kannon92) [SIG API Machinery]

• Updated API server internal API group to improve openapi schema correctness for fields being optional or required. (kubernetes/kubernetes#134675, @JoelSpeed) [SIG API Machinery, Apps, Auth, Node and Storage]

• Updated the /configz endpoint of kubelet, kube-scheduler, cloud controller manager, and kube-proxy to serialize the APIVersion and Kind fields and use public types instead of internal. (kubernetes/kubernetes#136044, @SergeyKanzhelev) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Network, Node, Scheduling and Testing]

• Add ResourcePoolStatusRequest API (v1alpha1) for querying DRA resource pool availability. This enables users and external schedulers to discover available devices across pools before submitting workloads. Requires the DRAResourcePoolStatus feature gate (alpha). (kubernetes/kubernetes#137028, @nmn3m) [SIG API Machinery, Apps, Auth, Etcd, Instrumentation, Node, Scheduling, Storage and Testing]

• Added DisruptionMode, PriorityClassName and Priority fields to Workload and PodGroup APIs to support workload-aware preemption when WorkloadAwarePreemption feature gate is enabled. (kubernetes/kubernetes#136589, @tosi3k) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• DRA (Dynamic Resource Allocation) drivers and controllers now require granular RBAC permissions to update ResourceClaim statuses when the DRAResourceClaimGranularStatusAuthorization feature gate is enabled (Beta in 1.36). Schedulers and controllers must be granted update/patch on resourceclaims/binding. DRA drivers must be granted associated-node:update or arbitrary-node:update (or patch equivalents) on resourceclaims/driver, restricted by their specific resourceNames. (kubernetes/kubernetes#134947, @aojea) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Scheduling and Testing]

• DRA: PodGroup resources can now make requests with ResourceClaims through a spec.resourceClaims field which can refer to ResourceClaims and ResourceClaimTemplates. Claims made by a PodGroup are reserved for the entire PodGroup instead of individual Pods, allowing more than 256 Pods to share a single ResourceClaim. ResourceClaimTemplates referenced by a PodGroup's claim will replicate into a ResourceClaim specific to that PodGroup able to be shared by all of the group's Pods. (kubernetes/kubernetes#136989, @nojnhuh) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Graduate InPlacePodLevelResourcesVerticalScaling feature to beta and have it on by default. This feature allows resizing the CPU and memory resources at pod-level for pods with pod-level resources set and enabled. (kubernetes/kubernetes#137684, @ndixita) [SIG API Machinery, Apps, Autoscaling, Node, Release, Scheduling and Testing]

• Promote NodeLogQuery to GA. (kubernetes/kubernetes#137544, @jrvaldes) [SIG Node and Windows]

• [Alpha] Introduce List Types for Attributes in DRA (KEP-5491).

  The DRAListTypeAttributes feature gate(false by default) can activate below enhancements.

  For DRA drivers, it can enable list-type fields(bools/ints/strings/versions) for device attributes in ResourceSlice. Please remember that the number of attribute values, including scalars and lists, per single device is limited to 48.

  For DRA users, this feature enhances the semantics of matchAttribute/distinctAttribute constraint in ResourceClaim to work on both scalar and list attributes. The matchAttribute constraint now matches when the intersection (as a set) of all the list values among candidate devices is non-empty. The distinctAttribute constraint, which is behind the ConsumableCapacity feature gate, matches when all the list values (as a set) among candidate devices are pairwise disjoint. In both constraints, scalar values are implicitly treated as a singleton set.

  And, a new CEL function .includes　is introduced. The function can work on both scalar and list attributes to test inclusion(e.g., device.attributes["dra.example.com"].model.includes("model-a")). This can support smooth migration for CEL expression in DRA resources when a DRA driver changes the attribute value type from scalar to list, or vice versa. (kubernetes/kubernetes#137190, @everpeace) [SIG API Machinery, Node, Scheduling and Testing]

• A few log calls which did not properly format their parameters were fixed. (kubernetes/kubernetes#137108, @pohly) [SIG API Machinery, Apps, Auth, Cluster Lifecycle, Network, Node, Scheduling and Testing]

• Add --tls-curve-preferences flag for configuring TLS key exchange mechanism (kubernetes/kubernetes#137115, @damdo) [SIG API Machinery, Architecture, CLI, Cloud Provider, Node and Testing]

• Add a deletion protection mechanism for PodGroup objects. (kubernetes/kubernetes#137641, @helayoty) [SIG API Machinery, Apps, Auth, Scheduling and Storage]

• Add admission plugin that validates PodGroup resources reference an existing Workload and match the declared PodGroupTemplate spec. (kubernetes/kubernetes#137464, @helayoty) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Add alpha support for manifest-based admission control configuration (KEP-5793). When the ManifestBasedAdmissionControlConfig feature gate is enabled, admission webhooks and CEL-based policies can be loaded from static manifest files on disk via the staticManifestsDir field in AdmissionConfiguration. These policies are active from API server startup, survive etcd unavailability, and can protect API-based admission resources from modification. (kubernetes/kubernetes#137346, @aramase) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage, Testing and Windows]

• Add opt-in alpha support in kubeletplugin framework for DRA drivers to publish DRA Device metadata in pod CDI mounts (kubernetes/kubernetes#137086, @alaypatel07) [SIG Apps, Network, Node and Testing]

• Add tlsServerName field to EgressSelectorConfiguration TLSConfig to allow overriding the server name used for TLS certificate verification (kubernetes/kubernetes#136640, @kennangaibel) [SIG API Machinery, Apps, Auth, Storage and Testing]

• Added MemoryReservationPolicy cgroup v2 MemoryQoS support to KubeletConfiguration for memory.min protection. (kubernetes/kubernetes#137584, @QiWang19) [SIG Node and Storage]

• Added PlacementGenerate extension point to the scheduler. It's used to generate placements for placement-based pod group scheduling. Its use is guarded by the TopologyAwareWorkloadScheduling feature gate. (kubernetes/kubernetes#137083, @brejman) [SIG Scheduling]

• Added PlacementScore extension point to the scheduler. It's used to score placements in order to choose the best one for placement-based pod group scheduling. Its use is guarded by the TopologyAwareWorkloadScheduling feature gate.

  Deprecated MinNodeScore and MaxNodeScore in favor of MinScore and MaxScore. (kubernetes/kubernetes#137201, @brejman) [SIG Scheduling]

• Added SchedulingConstraints to express TAS constraints for pod group scheduling behind TopologyAwareWorkloadScheduling feature gate.

  Added TopologyPlacement plugin implementing PlacementGenerate extension point to take the constraints into consideration during pod group scheduling. The usage of this plugin is guarded by the TopologyAwareWorkloadScheduling feature gate. (kubernetes/kubernetes#137271, @brejman) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Added TAS logic to the pod group scheduling cycle behind TopologyAwareWorkloadScheduling feature gate. This feature supports scheduling pod groups on nodes with matching topology domains. (kubernetes/kubernetes#137489, @brejman) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling and Testing]

• Added PodGroupPodsCount scheduler plugin to support workload-aware scheduling by prioritizing placements with higher pod counts within a group. (kubernetes/kubernetes#137488, @vshkrabkov) [SIG Scheduling and Testing]

• Added alpha support (behind PersistentVolumeClaimUnusedSinceTime feature gate) for tracking PVC unused status via a new Unused condition on PersistentVolumeClaimStatus. When enabled, the PVC protection controller sets Unused=True with a lastTransitionTime when no non-terminal Pods reference the PVC, enabling external automation to identify and manage unused storage. (kubernetes/kubernetes#137862, @gnufied) [SIG Apps, Auth, Storage and Testing]

• Added placement-based pod group scheduling algorithm to scheduler. Its use is guarded by the TopologyAwareWorkloadScheduling feature gate. (kubernetes/kubernetes#136944, @brejman) [SIG Scheduling and Testing]

• Allow users to opt-in to scheduling behaviour for CSI volume (kubernetes/kubernetes#137343, @gnufied) [SIG API Machinery, Scheduling and Storage]

• Config.k8s.io.flagz API is graduated to v1beta1 (kubernetes/kubernetes#137174, @richabanker) [SIG API Machinery, Instrumentation, Node, Scheduling and Testing]

• Config.k8s.io.statusz API is graduated to v1beta1 (kubernetes/kubernetes#137173, @richabanker) [SIG API Machinery, Instrumentation, Scheduling and Testing]

• DRA DeviceTaintRules: the TimeAdded of the taint is not only added automatically, it now also gets updated automatically when changing the effect. (kubernetes/kubernetes#137167, @pohly) [SIG API Machinery, Node and Testing]

• DRA extended resource feature is promoted to beta in 1.36 (kubernetes/kubernetes#135048, @yliaog) [SIG API Machinery, Architecture, Auth, Network, Node, Scheduling and Testing]

• DRA: graduate Device Binding Conditions (KEP #5007) to beta. The feature is now enabled by default in v1.36. (kubernetes/kubernetes#137795, @ttsuuubasa) [SIG API Machinery, Node, Scheduling and Testing]

• DRA: graduated device taints and tolerations (KEP #5055) to beta. Support for DeviceTaints in ResourceSlices is on by default. Support for DeviceTaintRules depends on enabling resource.k8s.io/v1beta2 and the DeviceTaintRules feature gate. (kubernetes/kubernetes#137170, @pohly) [SIG API Machinery, Apps, Auth, Cluster Lifecycle, Etcd, Node, Scheduling and Testing]

• DRAConsumableCapacity is enabled by default. (kubernetes/kubernetes#136611, @sunya-ch) [SIG API Machinery, Cluster Lifecycle, Node, Scheduling and Testing]

• Extended NodeResourcesFit to implement the PlacementScore extension point. The usage of the PlacementScore extension point is guarded by the TopologyAwareWorkloadScheduling feature gate. (kubernetes/kubernetes#136652, @brejman) [SIG Scheduling]

• Feature gate UserNamespacesSupport is now GA. (kubernetes/kubernetes#136792, @rata) [SIG API Machinery, Apps, CLI, Node, Storage and Testing]

• For pod resizes requested on nodes where the resize request exceeds the node's allocatable capacity or the node is running an OS that does not support resize, the request will now fail in admission rather than be marked as Infeasible in the pod status later. (kubernetes/kubernetes#136043, @natasha41575) [SIG API Machinery, Node, Release, Scheduling, Storage and Testing]

• Graduate metric 'apiserver_storage_events_received_total' to BETA (kubernetes/kubernetes#136314, @petern48) [SIG API Machinery, Etcd, Instrumentation and Testing]

• Graduated ImageVolume feature to stable. (kubernetes/kubernetes#136711, @saschagrunert) [SIG Apps, Architecture, Node and Testing]

• HPA: Improved scaling to and from zero with enabled HPAScaleToZero feature gate. (kubernetes/kubernetes#135118, @johanneswuerbach) [SIG Apps, Autoscaling and Testing]

• Integrate Workload and PodGroup APIs with the Job controllers to support gang-scheduling. (kubernetes/kubernetes#137032, @helayoty) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Instrumentation, Node, Scheduling and Testing]

• Introduced scheduling.k8s.io/v1alpha2 Workload and PodGroup API to allow for expressing workload-level scheduling requirements and let kube-scheduler act on those. Removed scheduling.k8s.io/v1alpha1 Workload API. (kubernetes/kubernetes#136976, @tosi3k) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Etcd, Node, Scheduling, Storage and Testing]

• Kube-scheduler now updates PodGroup status with a PodGroupScheduled condition reflecting whether the group was successfully scheduled or is unschedulable. (kubernetes/kubernetes#137611, @helayoty) [SIG API Machinery, Apps, Scheduling and Testing]

• Promote DRAPrioritizedList to GA (kubernetes/kubernetes#136924, @troychiu) [SIG Apps, Architecture, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Network, Node, Release, Scheduling, Storage and Testing]

• Promote ProcMountType feature to GA (kubernetes/kubernetes#137454, @haircommander) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing]

• Promote NodeDeclaredFeatures to beta. (kubernetes/kubernetes#136042, @pravk03) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation, Node, Scheduling, Storage and Testing]

• Promoted mutable CSI node allocatable count to GA. The MutableCSINodeAllocatableCount feature gate is now locked to enabled. (kubernetes/kubernetes#136230, @torredil) [SIG API Machinery and Storage]

• Promoted several endpointslice metrics from Alpha to Beta stability. (kubernetes/kubernetes#136368, @bhope) [SIG Instrumentation and Network]

• Promoted several scheduler metrics (scheduler_goroutines, scheduler_permit_wait_duration_seconds, scheduler_plugin_evaluation_total, scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds, scheduler_unschedulable_pods) from Alpha to Beta stability, providing stronger API and label stability guarantees for metric consumers. (kubernetes/kubernetes#136155, @bhope) [SIG Instrumentation and Scheduling]

• Promoted the DRAAdminAccess feature gate to GA. (kubernetes/kubernetes#137373, @ritazh) [SIG API Machinery, Auth, Node, Scheduling and Testing]

• Promoted two Job controller metrics from Alpha to Beta stability, providing stronger API and label stability guarantees for metric consumers. (kubernetes/kubernetes#136367, @bhope) [SIG Apps and Instrumentation]

• Remove CRD stored versions from status upon SVM migration (kubernetes/kubernetes#135297, @michaelasp) [SIG API Machinery, Apps, Auth and Testing]

• Renamed metric 'etcd_bookmark_counts' to 'etcd_bookmark_total'. If you are using custom monitoring dashboards or alerting rules based on the 'etcd_bookmark_counts' metric, please update them to use the new 'etcd_bookmark_total' metric. (kubernetes/kubernetes#136483, @petern48) [SIG API Machinery, Etcd, Instrumentation and Testing]

• Slow requests that use impersonation can now be tracked via the apiserver.latency.k8s.io/impersonation audit event annotation when the ConstrainedImpersonation feature is enabled. (kubernetes/kubernetes#137523, @enj) [SIG API Machinery, Auth and Testing]

• The /configz endpoint of kubelet, scheduler, cloud controller manager, and kube-proxy serializes the APIVersion and Kind fields as well as using public types instead of internal. (kubernetes/kubernetes#136044, @SergeyKanzhelev) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Network, Node, Scheduling and Testing]

• The ConstrainedImpersonation feature is now beta and enabled by default. (kubernetes/kubernetes#137609, @enj) [SIG API Machinery and Testing]

• The StrictIPCIDRValidation feature gate to kube-apiserver is now
  enabled by default, meaning that API fields no longer allow IP or CIDR
  values with extraneous leading "0"s (e.g., 010.000.000.005 rather than
  10.0.0.5) or CIDR subnet/mask values with ambiguous semantics (e.g.,
  192.168.0.5/24 rather than 192.168.0.0/24 or 192.168.0.5/32). (kubernetes/kubernetes#137053, @danwinship) [SIG Network and Testing]

• This change adds a new alpha feature DRANodeAllocatableResources, which includes:

  • A new ResourceSlice.Spec.Devices[*].NodeAllocatableResourceMappings field for DRA drivers to declare how device resources map to node allocatable Kubernetes resources (e.g., cpu, memory).
  • Changes in the DynamicResources plugin and the scheduler framework to correctly account for node allocatable resources requested through resource claims.
  • Kubelet's admission handler validates if the node can fulfill node allocatable DRA requests along with standard requests in the pod spec.
  text Copy

  #### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
  
  ([kubernetes/kubernetes#136725](https://github.com/kubernetes/kubernetes/pull/136725), [@pravk03](https://github.com/pravk03)) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• SnapshotMetadataService is now available in v1beta1 version. The support for the v1alpha1 version have been removed. (kubernetes/kubernetes#137564, @iPraveenParihar) [SIG Storage and Testing]

• Add a new concurrent-node-status-updates flag that is split from the concurrent-node-syncs flag (kubernetes/kubernetes#136716, @yonizxz) [SIG Cloud Provider]

• Fixed an issue in kube-apiserver, allowing it to recover from an established connection to an incorrect server that never returns the expected response during APIService availability checks. (kubernetes/kubernetes#137157, @bsalamat) [SIG API Machinery]

• Graduated MutatingAdmissionPolicy to GA (v1) in Kubernetes 1.36. The feature is now enabled by default. (kubernetes/kubernetes#136039, @lalitc375) [SIG API Machinery, Architecture, Etcd and Testing]

• Introduced stability-based lifecycle for declarative validation (Alpha/Beta/Stable). Scheduling Workload v1alpha1 now uses explicit declarative enforcement. (kubernetes/kubernetes#136793, @yongruilin) [SIG API Machinery and Scheduling]

• K8s.io/api: REST API types no longer implement marker ProtoMessage() methods, identifying them as standard v1 proto messages. Protobuf serialization of Kubernetes API types should use k8s.io/apimachinery/pkg/runtime/serializer/protobuf. See KEP-5589 for more details. (kubernetes/kubernetes#137084, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, Node, Scheduling and Storage]

• Pod Certificates (beta) now includes a PKCS#10 certificate signing request for wider compatibility with existing certificate authority software. (kubernetes/kubernetes#136729, @ahmedtd) [SIG API Machinery, Auth, Node and Testing]

• Promoted several component-base metrics (kubernetes_build_info, rest_client_requests_total, rest_client_request_duration_seconds, running_managed_controllers) from Alpha to Beta stability, providing stronger API and label stability guarantees for consumers. (kubernetes/kubernetes#136154, @bhope) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scalability, Scheduling, Storage and Testing]

• Replace deprecated sets.String with sets.Set[string] in apiserver admission subsystem. This is a breaking change for consumers of the NewLifecycle function. (kubernetes/kubernetes#134044, @mcallzbl) [SIG API Machinery and Auth]

• Updates API server internal API group to improve openapi schema correctness for fields being optional or required (kubernetes/kubernetes#134675, @JoelSpeed) [SIG API Machinery, Apps, Auth, Node and Storage]

• Add --concurrent-resourceclaim-syncs to configure kube-controller-manager resource claim reconcile concurrency (kubernetes/kubernetes#134701, @anson627) [SIG API Machinery, Apps, Node and Testing]

• Added negative duration validation for imageMinimumGCAge (kubernetes/kubernetes#135997, @ngopalak-redhat) [SIG API Machinery and Node]

• Clarified documentation and comments to indicate that the cpuCFSQuotaPeriod kubelet config field requires the CustomCPUCFSQuotaPeriod feature gate when using non-default values. No functional changes introduced. (kubernetes/kubernetes#133845, @rbiamru) [SIG Node and Release]

• Correct openapi schema union validation for the PodGroupPolicy struct in scheduling v1alpha1 (kubernetes/kubernetes#136424, @JoelSpeed) [SIG API Machinery and Scheduling]

• Fixed a potential nil pointer dereference in the scheduler's NodeResourcesFitArgs validation when using RequestedToCapacityRatio scoring strategy (kubernetes/kubernetes#132120, @flpanbin) [SIG Scheduling]

• Fixes fake.NewClientset() to work properly with correct schema. (kubernetes/kubernetes#131068, @soltysh) [SIG API Machinery]

• Generate fake.NewClientset which replace the deprecated NewSimpleClientset, for kube-aggregator and sample-apiserver (kubernetes/kubernetes#136537, @soltysh) [SIG API Machinery]

• Graduate watch_list_duration_seconds from ALPHA to BETA (kubernetes/kubernetes#136086, @richabanker) [SIG API Machinery, Instrumentation, Node and Testing]

• Kube-apiserver: the --audit-policy-file config file now supports specifying group: "*" in resources rules to match all API groups (kubernetes/kubernetes#135262, @cmuuss) [SIG API Machinery, Auth and Testing]

• Kube-controller-manager: alpha gauge metrics for informer queue length are now published as informer_queued_items{name=kube-controller-manager,group=<group>,resource=<resource>,version=<version>} <count> (kubernetes/kubernetes#135782, @richabanker) [SIG API Machinery, Architecture, Instrumentation and Testing]

• Locked the feature-gate VolumeAttributesClass to default (true) and bump VolumeAttributesClass preferred storage version to storage.k8s.io/v1 (kubernetes/kubernetes#134556, @carlory) [SIG API Machinery, Apps, Etcd, Network, Node, Scheduling, Storage and Testing]

• Promote workqueue metrics from ALPHA to BETA (kubernetes/kubernetes#135522, @petern48) [SIG Architecture, Instrumentation and Testing]

• Removed the generally available feature gate CSIMigrationPortworx, which was locked and enabled since 1.33.

  • Removed alpha feature gate InTreePluginPortworxUnregister
  • Removed Portworx volume plugin from in-tree plugins because all operations are redirected to CSI. (kubernetes/kubernetes#135322, @carlory) [SIG API Machinery, Apps, Auth, Node, Scalability, Scheduling, Storage and Testing]

• The ImageVolumeWithDigest is added which adds the digest of image volumes to the container's status. (kubernetes/kubernetes#132807, @iholder101) [SIG API Machinery, Apps, Node and Testing]

• The endpoints field in discovery.k8s.io/v1 EndpointSlice is now correctly defined as optional in the OpenAPI specification, matching the server's behavior. (kubernetes/kubernetes#136111, @aojea) [SIG Network]

• Update API comments to reflect that stable state of Dynamic Resource Allocation (kubernetes/kubernetes#136441, @kannon92) [SIG API Machinery]

Basic asyncio support (Experimental)

• Asyncio package is experimental currently, breaking changes may be introduced
  in future releases.

• Basic asyncio with kube config, and in_cluster_config is added.

• Dynamic client, watch, stream, shared informer, leader election are not yet supported in asyncio.

Breaking Change from upgrading openapi generator to v6.6.0

• Legacy dict(str, str) syntax is no longer supported in ApiClient deserializer. Only modern dict[str, str] syntax is supported.

• All models init will now use Configuration.get_default_copy() instead of Configuration() if the passed in parameter local_vars_configuration is None. OpenAPITools/openapi-generator#8500.

• ApiClient's update_params_for_auth method has one parameter name changed from querys to queries.

• Configuration auth uses 'BearerToken' instead of 'authorization' in api_key.

• V1ServiceSpec's external_i_ps has been renamed to external_ips, and
  cluster_i_ps to cluster_ips.

v35.0.0

Kubernetes API Version: v1.35.0

v35.0.0b1

Kubernetes API Version: v1.35.0

v35.0.0a1

Kubernetes API Version: v1.35.0

API Change Copy Section

• Added ObservedGeneration to CustomResourceDefinition conditions. (kubernetes/kubernetes#134984, @michaelasp)

• Added WithOrigin within apis/core/validation with adjusted tests. (kubernetes/kubernetes#132825, @PatrickLaabs)

• Added scoring for the prioritized list feature so nodes that best satisfy the highest-ranked subrequests were chosen. (kubernetes/kubernetes#134711, @mortent) [SIG Node, Scheduling and Testing]

• Added the --min-compatibility-version flag to kube-apiserver, kube-controller-manager, and kube-scheduler. (kubernetes/kubernetes#133980, @siyuanfoundation) [SIG API Machinery, Architecture, Cluster Lifecycle, Etcd, Scheduling and Testing]

• Added the StorageVersionMigration v1beta1 API and removed the v1alpha1 API.

  ACTION REQUIRED: The v1alpha1 API is no longer supported. Users must remove any v1alpha1 resources before upgrading. (kubernetes/kubernetes#134784, @michaelasp) [SIG API Machinery, Apps, Auth, Etcd and Testing]

• Added validation to ensure log-flush-frequency is a positive value, returning an error instead of causing a panic. (kubernetes/kubernetes#133540, @BenTheElder) [SIG Architecture, Instrumentation, Network and Node]

• All containers are restarted when a source container in a restart policy rule exits. This alpha feature is gated behind RestartAllContainersOnContainerExit. (kubernetes/kubernetes#134345, @yuanwang04) [SIG Apps, Node and Testing]

• CSI drivers can now opt in to receive service account tokens via the secrets field instead of volume context by setting spec.serviceAccountTokenInSecrets: true in the CSIDriver object. This prevents tokens from being exposed in logs and other outputs. The feature is gated by the CSIServiceAccountTokenSecrets feature gate (beta in v1.35). (kubernetes/kubernetes#134826, @aramase) [SIG API Machinery, Auth, Storage and Testing]

• Changed kuberc configuration schema. Two new optional fields added to kuberc configuration, credPluginPolicy and credPluginAllowlist. This is documented in KEP-3104 and documentation is added to the website by kubernetes/website#52877 (kubernetes/kubernetes#134870, @pmengelbert) [SIG API Machinery, Architecture, Auth, CLI, Instrumentation and Testing]

• DRA device taints: DeviceTaintRule status provides information about the rule, including whether Pods still need to be evicted (EvictionInProgress condition). The newly added None effect can be used to preview what a DeviceTaintRule would do if it used the NoExecute effect and to taint devices (device health) without immediately affecting scheduling or running Pods. (kubernetes/kubernetes#134152, @pohly) [SIG API Machinery, Apps, Auth, Node, Release, Scheduling and Testing]

• DRA: The DynamicResourceAllocation feature gate for the core functionality (GA in v1.34) has now been locked to enabled-by-default and cannot be disabled anymore. (kubernetes/kubernetes#134452, @pohly) [SIG Auth, Node, Scheduling and Testing]

• Enabled kubectl get -o kyaml by default. To disable it, set KUBECTL_KYAML=false. (kubernetes/kubernetes#133327, @thockin)

• Enabled in-place resizing of pod-level resources.

  • Added Resources in PodStatus to capture resources set in the pod-level cgroup.
  • Added AllocatedResources in PodStatus to capture resources requested in the PodSpec. (kubernetes/kubernetes#132919, @ndixita) [SIG API Machinery, Apps, Architecture, Auth, CLI, Instrumentation, Node, Scheduling and Testing]

• Enabled the NominatedNodeNameForExpectation feature in kube-scheduler by default.

  • Enabled the ClearingNominatedNodeNameAfterBinding feature in kube-apiserver by default. (kubernetes/kubernetes#135103, @ania-borowiec) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• Enhanced discovery responses to merge API groups and resources from all peer apiservers when the UnknownVersionInteroperabilityProxy feature is enabled. (kubernetes/kubernetes#133648, @richabanker) [SIG API Machinery, Auth, Cloud Provider, Node, Scheduling and Testing]

• Extended core/v1 Toleration to support numeric comparison operators (Gt,Lt). (kubernetes/kubernetes#134665, @helayoty) [SIG API Machinery, Apps, Node, Scheduling, Testing and Windows]

• Feature gate dependencies are now explicit, and validated at startup. A feature can no longer be enabled if it depends on a disabled feature. In particular, this means that AllAlpha=true will no longer work without enabling disabled-by-default beta features that are depended on (either with AllBeta=true or explicitly enumerating the disabled dependencies). (kubernetes/kubernetes#133697, @tallclair) [SIG API Machinery, Architecture, Cluster Lifecycle and Node]

• Generated OpenAPI model packages for API types into zz_generated.model_name.go files, accessible via the OpenAPIModelName() function. This allows API authors to declare desired OpenAPI model packages instead of relying on the Go package path of API types. (kubernetes/kubernetes#131755, @jpbetz) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• Implemented constrained impersonation as described in KEP-5284. (kubernetes/kubernetes#134803, @enj) [SIG API Machinery, Auth and Testing]

• Introduced a new declarative validation tag +k8s:customUnique to control listmap uniqueness. (kubernetes/kubernetes#134279, @yongruilin) [SIG API Machinery and Auth]

• Introduced a structured and versioned v1alpha1 response for the statusz endpoint. (kubernetes/kubernetes#134313, @richabanker) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

• Introduced a structured and versioned v1alpha1 response format for the flagz endpoint. (kubernetes/kubernetes#134995, @yongruilin) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

• Introduced the GangScheduling kube-scheduler plugin to support "all-or-nothing" scheduling using the scheduling.k8s.io/v1alpha1 Workload API. (kubernetes/kubernetes#134722, @macsko) [SIG API Machinery, Apps, Auth, CLI, Etcd, Scheduling and Testing]

• Introduced the Node Declared Features capability (alpha), which includes:

  • A new Node.Status.DeclaredFeatures field for publishing node-specific features.
  • A component-helpers library for feature registration and inference.
  • A NodeDeclaredFeatures scheduler plugin to match pods with nodes that provide required features.
  • A NodeDeclaredFeatureValidator admission plugin to validate pod updates against a node's declared features. (kubernetes/kubernetes#133389, @pravk03) [SIG API Machinery, Apps, Node, Release, Scheduling and Testing]

• Introduced the scheduling.k8s.io/v1alpha1 Workload API to express workload-level scheduling requirements and allow the kube-scheduler to act on them. (kubernetes/kubernetes#134564, @macsko) [SIG API Machinery, Apps, CLI, Etcd, Scheduling and Testing]

• Introduced the alpha MutableSchedulingDirectivesForSuspendedJobs feature gate (disabled by default), which allows mutating a Job's scheduling directives while the Job is suspended.
  It also updates the Job controller to clears the status.startTime field for suspended Jobs. (kubernetes/kubernetes#135104, @mimowo) [SIG Apps and Testing]

• Kube-apiserver: Fixed a v1.34 regression in CustomResourceDefinition handling that incorrectly warned about unrecognized formats on number and integer properties. (kubernetes/kubernetes#133896, @yongruilin) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Contributor Experience, Network, Node and Scheduling]

• Kube-apiserver: Fixed a possible panic validating a custom resource whose CustomResourceDefinition indicates a status subresource exists, but which does not define a status property in the openAPIV3Schema. (kubernetes/kubernetes#133721, @fusida) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]

• Kubernetes API Go types removed runtime use of the github.com/gogo/protobuf library, and are no longer registered into the global gogo type registry. Kubernetes API Go types were not suitable for use with the google.golang.org/protobuf library, and no longer implement ProtoMessage() by default to avoid accidental incompatible use. If removal of these marker methods impacts your use, it can be re-enabled for one more release with a kubernetes_protomessage_one_more_release build tag, but will be removed in v1.36. (kubernetes/kubernetes#134256, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling and Storage]

• Made node affinity in Persistent Volume mutable. (kubernetes/kubernetes#134339, @huww98) [SIG API Machinery, Apps and Node]

• Moved the ImagePullIntent and ImagePulledRecord objects used by the kubelet to track image pulls to the v1beta1 API version. (kubernetes/kubernetes#132579, @stlaz) [SIG Auth and Node]

• Pod resize now only allows CPU and memory resources; other resource types are forbidden. (kubernetes/kubernetes#135084, @tallclair) [SIG Apps, Node and Testing]

• Prevented Pods from being scheduled onto nodes that lack the required CSI driver. (kubernetes/kubernetes#135012, @gnufied) [SIG API Machinery, Scheduling, Storage and Testing]

• Promoted HPA configurable tolerance to beta. The HPAConfigurableTolerance feature gate has now been enabled by default. (kubernetes/kubernetes#133128, @jm-franc) [SIG API Machinery and Autoscaling]

• Promoted ReplicaSet and Deployment .status.terminatingReplicas tracking to beta. The DeploymentReplicaSetTerminatingReplicas feature gate is now enabled by default. (kubernetes/kubernetes#133087, @atiratree) [SIG API Machinery, Apps and Testing]

• Promoted PodObservedGenerationTracking to GA. (kubernetes/kubernetes#134948, @natasha41575) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Promoted the JobManagedBy feature to general availability. The JobManagedBy feature gate was locked to true and will be removed in a future Kubernetes release. (kubernetes/kubernetes#135080, @dejanzele) [SIG API Machinery, Apps and Testing]

• Promoted the MaxUnavailableStatefulSet feature to beta and enabling it by default. (kubernetes/kubernetes#133153, @helayoty) [SIG API Machinery and Apps]

• Removed the StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks feature gates, which were locked since v1.32. (kubernetes/kubernetes#134994, @liggitt) [SIG API Machinery, Auth, Node and Testing]

• Scheduler: Added the bindingTimeout argument to the DynamicResources plugin configuration, allowing customization of the wait duration in PreBind for device binding conditions.
  Defaults to 10 minutes when DRADeviceBindingConditions and DRAResourceClaimDeviceStatus are both enabled. (kubernetes/kubernetes#134905, @fj-naji) [SIG Node and Scheduling]

• The DRA device taints and toleration feature received a separate feature gate, DRADeviceTaintRules, which controlled support for DeviceTaintRules. This allowed disabling it while keeping DRADeviceTaints enabled so that tainting via ResourceSlices continued to work. (kubernetes/kubernetes#135068, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• The Pod Certificates feature moved to beta. The PodCertificateRequest feature gate is set disabled by default. To use the feature, users must enable the certificates API groups in v1beta1 and enable the PodCertificateRequest feature gate. The UserAnnotations field was added to the PodCertificateProjection API and the corresponding UnverifiedUserAnnotations field was added to the PodCertificateRequest API. (kubernetes/kubernetes#134624, @yt2985) [SIG API Machinery, Apps, Auth, Etcd, Instrumentation, Node and Testing]

• The KubeletEnsureSecretPulledImages feature was promoted to Beta and enabled by default. (kubernetes/kubernetes#135228, @aramase) [SIG Auth, Node and Testing]

• The PreferSameZone and PreferSameNode values for the Service
  trafficDistribution field graduated to general availability. The
  PreferClose value is now deprecated in favor of the more explicit
  PreferSameZone. (kubernetes/kubernetes#134457, @danwinship) [SIG API Machinery, Apps, Network and Testing]

• Updated ResourceQuota to count device class requests within a ResourceClaim as two additional quotas when the DRAExtendedResource feature is enabled:

  • requests.deviceclass.resource.k8s.io/<deviceclass> is charged based on the worst-case number of devices requested.
  • Device classes mapping to an extended resource now consume requests.<extended resource name>. (kubernetes/kubernetes#134210, @yliaog) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Updated storage version for MutatingAdmissionPolicy to v1beta1. (kubernetes/kubernetes#133715, @cici37) [SIG API Machinery, Etcd and Testing]

• Updated the Partitionable Devices feature to support referencing counter sets across ResourceSlices within the same resource pool. Devices from incomplete pools were no longer considered for allocation. This change introduced backwards-incompatible updates to the alpha feature, requiring any ResourceSlices using it to be removed before upgrading or downgrading between v1.34 and v1.35. (kubernetes/kubernetes#134189, @mortent) [SIG API Machinery, Node, Scheduling and Testing]

• Upgraded the PodObservedGenerationTracking feature to beta in v1.34 and removed the alpha version description from the OpenAPI specification. (kubernetes/kubernetes#133883, @yangjunmyfm192085)

• Add scoring for the prioritized list feature so that the node that can satisfy the best ranked subrequests are chosen. (kubernetes/kubernetes#134711, @mortent) [SIG Node, Scheduling and Testing]

• Allows restart all containers when the source container exits with a matching restart policy rule. This is an alpha feature behind feature gate RestartAllContainersOnContainerExit. (kubernetes/kubernetes#134345, @yuanwang04) [SIG Apps, Node and Testing]

• Changed kuberc configuration schema. Two new optional fields added to kuberc configuration, credPluginPolicy and credPluginAllowlist. This is documented in KEP-3104 and documentation is added to the website by kubernetes/website#52877 (kubernetes/kubernetes#134870, @pmengelbert) [SIG API Machinery, Architecture, Auth, CLI, Instrumentation and Testing]

• Enhanced discovery response to support merged API groups/resources from all peer apiservers when UnknownVersionInteroperabilityProxy feature is enabled (kubernetes/kubernetes#133648, @richabanker) [SIG API Machinery, Auth, Cloud Provider, Node, Scheduling and Testing]

• Extend core/v1 Toleration to support numeric comparison operators (Gt, Lt). (kubernetes/kubernetes#134665, @helayoty) [SIG API Machinery, Apps, Node, Scheduling, Testing and Windows]

• Features: NominatedNodeNameForExpectation in kube-scheduler and CleaeringNominatedNodeNameAfterBinding in kube-apiserver are now enabled by default. (kubernetes/kubernetes#135103, @ania-borowiec) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• Implement changes to prevent pod scheduling to a node without CSI driver (kubernetes/kubernetes#135012, @gnufied) [SIG API Machinery, Scheduling, Storage and Testing]

• Introduce scheduling.k8s.io/v1alpha1 Workload API to allow for expressing workload-level scheduling requirements and let kube-scheduler act on those. (kubernetes/kubernetes#134564, @macsko) [SIG API Machinery, Apps, CLI, Etcd, Scheduling and Testing]

• Introduce the alpha MutableSchedulingDirectivesForSuspendedJobs feature gate (disabled by default) which:

  1. allows to mutate Job's scheduling directives for suspended Jobs
  2. makes the Job controller to clear the status.startTime field for suspended Jobs (kubernetes/kubernetes#135104, @mimowo) [SIG Apps and Testing]

• Introduced GangScheduling kube-scheduler plugin to enable "all-or-nothing" scheduling. Workload API in scheduling.k8s.io/v1alpha1 is used to express the desired policy. (kubernetes/kubernetes#134722, @macsko) [SIG API Machinery, Apps, Auth, CLI, Etcd, Scheduling and Testing]

• PV node affinity is now mutable. (kubernetes/kubernetes#134339, @huww98) [SIG API Machinery, Apps and Node]

• ResourceQuota now counts device class requests within a ResourceClaim object as consuming two additional quotas when the DRAExtendedResource feature is enabled:

  • requests.deviceclass.resource.k8s.io/<deviceclass> with a quantity equal to the worst case count of devices requested
  • requests for device classes that map to an extended resource consume requests.<extended resource name> (kubernetes/kubernetes#134210, @yliaog) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• The DRA device taints and toleration feature now has a separate feature gate, DRADeviceTaintRules, which controls whether support for DeviceTaintRules is enabled. It is possible to disable that and keep DRADeviceTaints enabled, in which case tainting by DRA drivers through ResourceSlices continues to work. (kubernetes/kubernetes#135068, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• The ImagePullIntent and ImagePulledRecord objects used by kubelet to store information about image pulls have been moved to the v1beta1 API version. (kubernetes/kubernetes#132579, @stlaz) [SIG Auth and Node]

• The KubeletEnsureSecretPulledImages feature is now beta and enabled by default. (kubernetes/kubernetes#135228, @aramase) [SIG Auth, Node and Testing]

• This change adds a new alpha feature Node Declared Features, which includes:

  • A new Node.Status.DeclaredFeatures field for Kubelet to publish node-specific features.
  • A library in component-helpers for feature registration and inference.
  • A scheduler plugin (NodeDeclaredFeatures) scheduler plugin to match pods with nodes that provide their required features.
  • An admission plugin (NodeDeclaredFeatureValidator) to validate pod updates against a node's declared features. (kubernetes/kubernetes#133389, @pravk03) [SIG API Machinery, Apps, Node, Release, Scheduling and Testing]

• This change allows In Place Resize of Pod Level Resources

  • Add Resources in PodStatus to capture resources set at pod-level cgroup
  • Add AllocatedResources in PodStatus to capture resources requested in the PodSpec (kubernetes/kubernetes#132919, @ndixita) [SIG API Machinery, Apps, Architecture, Auth, CLI, Instrumentation, Node, Scheduling and Testing]

• Updates to the Partitionable Devices feature which allows for referencing counter sets across different ResourceSlices within the same resource pool.

  Devices from incomplete pools are no longer considered for allocation.

  This contains backwards incompatible changes to the Partitionable Devices alpha feature, so any ResourceSlices that uses the feature should be removed prior to upgrading or downgrading between 1.34 and 1.35. (kubernetes/kubernetes#134189, @mortent) [SIG API Machinery, Node, Scheduling and Testing]

• Add ObservedGeneration to CustomResourceDefinition Conditions. (kubernetes/kubernetes#134984, @michaelasp) [SIG API Machinery]

• Add StorageVersionMigration v1beta1 api and remove the v1alpha API.

  Any use of the v1alpha1 api is no longer supported and
  users must remove any v1alpha1 resources prior to upgrade. (kubernetes/kubernetes#134784, @michaelasp) [SIG API Machinery, Apps, Auth, Etcd and Testing]

• CSI drivers can now opt-in to receive service account tokens via the secrets field instead of volume context by setting spec.serviceAccountTokenInSecrets: true in the CSIDriver object. This prevents tokens from being exposed in logs and other outputs. The feature is gated by the CSIServiceAccountTokenSecrets feature gate (Beta in v1.35). (kubernetes/kubernetes#134826, @aramase) [SIG API Machinery, Auth, Storage and Testing]

• DRA device taints: DeviceTaintRule status provided information about the rule, in particular whether pods still need to be evicted ("EvictionInProgress" condition). The new "None" effect can be used to preview what a DeviceTaintRule would do if it used the "NoExecute" effect and to taint devices ("device health") without immediately affecting scheduling or running pods. (kubernetes/kubernetes#134152, @pohly) [SIG API Machinery, Apps, Auth, Node, Release, Scheduling and Testing]

• DRA: the DynamicResourceAllocation feature gate for the core functionality (GA in 1.34) is now locked to enabled-by-default and thus cannot be disabled anymore. (kubernetes/kubernetes#134452, @pohly) [SIG Auth, Node, Scheduling and Testing]

• Forbid adding resources other than CPU & memory on pod resize. (kubernetes/kubernetes#135084, @tallclair) [SIG Apps, Node and Testing]

• Implement constrained impersonation as described in https://kep.k8s.io/5284 (kubernetes/kubernetes#134803, @enj) [SIG API Machinery, Auth and Testing]

• Introduces a structured and versioned v1alpha1 response for flagz (kubernetes/kubernetes#134995, @yongruilin) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

• Introduces a structured and versioned v1alpha1 response for statusz (kubernetes/kubernetes#134313, @richabanker) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

• New --min-compatibility-version flag for apiserver, kcm and kube scheduler (kubernetes/kubernetes#133980, @siyuanfoundation) [SIG API Machinery, Architecture, Cluster Lifecycle, Etcd, Scheduling and Testing]

• Promote PodObservedGenerationTracking to GA. (kubernetes/kubernetes#134948, @natasha41575) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Promoted Job Managed By to general availability. The JobManagedBy feature gate is now locked to true, and will be removed in a future release of Kubernetes. (kubernetes/kubernetes#135080, @dejanzele) [SIG API Machinery, Apps and Testing]

• Promoted ReplicaSet and Deployment .status.terminatingReplicas tracking to beta. The DeploymentReplicaSetTerminatingReplicas feature gate is now enabled by default. (kubernetes/kubernetes#133087, @atiratree) [SIG API Machinery, Apps and Testing]

• Scheduler: added a new bindingTimeout argument to the DynamicResources plugin configuration.
  This allows customizing the wait duration in PreBind for device binding conditions.
  Defaults to 10 minutes when DRADeviceBindingConditions and DRAResourceClaimDeviceStatus are both enabled. (kubernetes/kubernetes#134905, @fj-naji) [SIG Node and Scheduling]

• The Pod Certificates feature is moving to beta. The PodCertificateRequest feature gate is still set false by default. To use the feature, users will need to enable the certificates API groups in v1beta1 and enable the feature gate PodCertificateRequest. A new field UserAnnotations is added to the PodCertificateProjection API and the corresponding UnverifiedUserAnnotations is added to the PodCertificateRequest API. (kubernetes/kubernetes#134624, @yt2985) [SIG API Machinery, Apps, Auth, Etcd, Instrumentation, Node and Testing]

• The StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks feature gates, locked on since 1.32, have been removed (kubernetes/kubernetes#134994, @liggitt) [SIG API Machinery, Auth, Node and Testing]

• The PreferSameZone and PreferSameNode values for Service's
  trafficDistribution field are now GA. The old value PreferClose is now
  deprecated in favor of the more-explicit PreferSameZone. (kubernetes/kubernetes#134457, @danwinship) [SIG API Machinery, Apps, Network and Testing]

• Kube-apiserver: fix a possible panic validating a custom resource whose CustomResourceDefinition indicates a status subresource exists, but which does not define a status property in the openAPIV3Schema (kubernetes/kubernetes#133721, @fusida) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]

• Kubernetes API Go types removed runtime use of the github.com/gogo/protobuf library, and are no longer registered into the global gogo type registry. Kubernetes API Go types were not suitable for use with the google.golang.org/protobuf library, and no longer implement ProtoMessage() by default to avoid accidental incompatible use. If removal of these marker methods impacts your use, it can be re-enabled for one more release with a kubernetes_protomessage_one_more_release build tag, but will be removed in 1.36. (kubernetes/kubernetes#134256, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling and Storage]

• Promoted HPA configurable tolerance to beta. The HPAConfigurableTolerance feature gate is now enabled by default. (kubernetes/kubernetes#133128, @jm-franc) [SIG API Machinery and Autoscaling]

• The MaxUnavailableStatefulSet feature is now beta and enabled by default. (kubernetes/kubernetes#133153, @helayoty) [SIG API Machinery and Apps]

• Added WithOrigin within apis/core/validation with adjusted tests (kubernetes/kubernetes#132825, @PatrickLaabs) [SIG Apps]

• Component-base: validate that log-flush-frequency is positive and return an error instead of panic-ing (kubernetes/kubernetes#133540, @BenTheElder) [SIG Architecture, Instrumentation, Network and Node]

• Feature gate dependencies are now explicit, and validated at startup. A feature can no longer be enabled if it depends on a disabled feature. In particular, this means that AllAlpha=true will no longer work without enabling disabled-by-default beta features that are depended on (either with AllBeta=true or explicitly enumerating the disabled dependencies). (kubernetes/kubernetes#133697, @tallclair) [SIG API Machinery, Architecture, Cluster Lifecycle and Node]

• In version 1.34, the PodObservedGenerationTracking feature has been upgraded to beta, and the description of the alpha version in the openapi has been removed. (kubernetes/kubernetes#133883, @yangjunmyfm192085) [SIG Apps]

• Introduce a new declarative validation tag +k8s:customUnique to control listmap uniqueness (kubernetes/kubernetes#134279, @yongruilin) [SIG API Machinery and Auth]

• Kube-apiserver: Fixed a 1.34 regression in CustomResourceDefinition handling that incorrectly warned about unrecognized formats on number and integer properties (kubernetes/kubernetes#133896, @yongruilin) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Contributor Experience, Network, Node and Scheduling]

• OpenAPI model packages of API types are generated into zz_generated.model_name.go files and are accessible using the OpenAPIModelName() function. This allows API authors to declare the desired OpenAPI model packages instead of using the go package path of API types. (kubernetes/kubernetes#131755, @jpbetz) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• Support for kubectl get -o kyaml is now on by default. To disable it, set KUBECTL_KYAML=false. (kubernetes/kubernetes#133327, @thockin) [SIG CLI]

• The storage version for MutatingAdmissionPolicy is updated to v1beta1. (kubernetes/kubernetes#133715, @cici37) [SIG API Machinery, Etcd and Testing]

v34.1.0

Kubernetes API Version: v1.34.1

v34.1.0b1

Kubernetes API Version: v1.34.1

v34.1.0a1

Kubernetes API Version: v1.34.1

API Change Copy Section

• Added omitempty and opt tag to the API v1beta2 AdminAccess type in the DeviceRequestAllocationResult struct. (kubernetes/kubernetes#132338, @PatrickLaabs)

• Added a runtime.ApplyConfiguration interface implemented by all generated apply configuration types. (kubernetes/kubernetes#132194, @alvaroaleman) [SIG API Machinery and Instrumentation]

• Added a detailed event for in-place pod vertical scaling completed, improving cluster management and debugging. (kubernetes/kubernetes#130387, @shiya0705) [SIG API Machinery, Apps, Autoscaling, Node, Scheduling and Testing]

• Added a mechanism for configurable container restarts: container-level restart rules. This was an alpha feature behind the ContainerRestartRules feature gate. (kubernetes/kubernetes#132642, @yuanwang04) [SIG API Machinery, Apps, Node and Testing]

• Added a new FileKeyRef field to containers, allowing them to load variables from files by setting this field.

  Introduced the EnvFiles feature gate to govern activation of this functionality. (kubernetes/kubernetes#132626, @HirazawaUi) [SIG API Machinery, Apps, Node and Testing]

• Added driver-owned fields in ResourceSlice to mark whether the device was shareable among multiple resource claims (or requests) and to specify how each capacity could be shared between different requests.

  • Added user-owned fields in ResourceClaim to specify resource requirements against each device capacity.
  • Added scheduler-owned field in ResourceClaim.Status to specify how much device capacity is reserved for a specific request.
  • Added an additional identifier to ResourceClaim.Status for the device supports multiple allocations.
  • Added a new constraint type to enforce uniqueness of specified attributes across all allocated devices. (kubernetes/kubernetes#132522, @sunya-ch) [SIG API Machinery, Apps, Architecture, CLI, Cluster Lifecycle, Network, Node, Release, Scheduling and Testing]

• Added new optional APIs in ResouceSlice.Basic and ResourceClaim.Status.AllocatedDeviceStatus. (kubernetes/kubernetes#130160, @KobayashiD27) [SIG API Machinery, Apps, Architecture, Node, Release, Scheduling and Testing]

• Added support for specifying controlplane or cluster egress selectors in JWT authenticators via the issuer.egressSelectorType field in the AuthenticationConfiguration.jwt array. If unset, the previous behavior of using no egress selector is preserved. This functionality requires the StructuredAuthenticationConfigurationEgressSelector beta feature gate (enabled by default). (kubernetes/kubernetes#132768, @enj) [SIG API Machinery, Auth and Testing]

• Added support in the Kubelet for monitoring the health of devices allocated via Dynamic Resource Allocation (DRA) and report it in the pod.status.containerStatuses.allocatedResourcesStatus field. This required the DRA plugin to implement the new v1alpha1 NodeHealth gRPC service. This feature was controlled by the ResourceHealthStatus feature gate. (kubernetes/kubernetes#130606, @Jpsassine) [SIG Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Network, Node, Release, Scheduling, Storage and Testing]

• Added support in the kubelet's image pull credential tracking for service account-based verification. When an image was pulled using service account credentials via external credential providers, subsequent Pods using the same service account (UID, name, and namespace) could access the cached image without re-authentication for the lifetime of that service account. (kubernetes/kubernetes#132771, @aramase) [SIG Auth, Node and Testing]

• Added validation to reject Pods using the PodLevelResources feature on Windows OS due to lack of support. The API server rejected Pods with pod-level resources and a Pod.spec.os.name targeting Windows. Kubelet on nodes running Windows also rejected Pods with pod-level resources at the admission phase. (kubernetes/kubernetes#133046, @toVersus) [SIG Apps and Node]

• Added warnings when creating headless service with set loadBalancerIP,externalIPs and/or SessionAffinity. (kubernetes/kubernetes#132214, @Peac36)

• Allowed pvc.spec.VolumeAttributesClassName to change from non-nil to nil. (kubernetes/kubernetes#132106, @AndrewSirenko)

• Allowed setting the hostnameOverride field in PodSpec to specify any RFC 1123 DNS subdomain as the pod's hostname. The HostnameOverride feature gate was introduced to control enablement of this functionality. (kubernetes/kubernetes#132558, @HirazawaUi) [SIG API Machinery, Apps, Network, Node and Testing]

• Changed underlying logic for Eviction Manager helper functions. (kubernetes/kubernetes#132277, @KevinTMtz) [SIG Node, Scheduling and Testing]

• Changed underlying logic to propagate pod-level hugepage cgroup to containers when they did not specify hugepage resources.

  • Added validation to enforce the hugepage aggregated container limits to be smaller than or equal to pod-level limits. This was already enforced with the defaulted requests from the specified limits, however it did not make it clear about both hugepage requests and limits. (kubernetes/kubernetes#131089, @KevinTMtz) [SIG Apps, Node and Testing]

• Corrected the documentation to clarify that podSelector is optional and described its default behavior. (kubernetes/kubernetes#131354, @tomoish)

• DRA API: resource.k8s.io/v1alpha3 now only contains DeviceTaintRule. All other types got removed because they became obsolete when introducing the v1beta1 API in 1.32.
  before updating a cluster where resourceclaims, resourceclaimtemplates, deviceclasses, or resourceslices might have been stored using Kubernetes < 1.32, delete all of those resources before updating and recreate them as needed while running Kubernetes >= 1.32. (kubernetes/kubernetes#132000, @pohly) [SIG Etcd, Node, Scheduling and Testing]

• DRA: Starting with Kubernetes 1.34, the alpha-level resource.k8s.io/admin-access label has been updated to resource.kubernetes.io/admin-access. Admins using the alpha feature and updating from 1.33 can set both labels, upgrade, then remove resource.k8s.io/admin-access when no downgrade is going to happen anymore. (kubernetes/kubernetes#131996, @ritazh) [SIG Node and Testing]

• DRA: The scheduler plugin prevented abnormal filter runtimes by timing out after 10 seconds. This was configurable via the plugin configuration's FilterTimeout. Setting it to zero disabled the timeout and restored the behavior of Kubernetes <= 1.33. (kubernetes/kubernetes#132033, @pohly) [SIG Node, Scheduling and Testing]

• DRA: When the prioritized list feature was used in a request and the resulting number of allocated devices exceeded the number of allowed devices per claim, the scheduler aborted the attempt to allocate devices early. Previously, it tried to many different combinations, which could take a long time. (kubernetes/kubernetes#130593, @mortent) [SIG Apps, Node, Scheduling and Testing]

• DRA: removed support for the v1alpha4 kubelet gRPC API (added in 1.31, superseded in 1.32). DRA drivers using the helper package from Kubernetes >= 1.32 use the v1beta1 API and continue to be supported. (kubernetes/kubernetes#132574, @pohly)

• Deprecated StreamingConnectionIdleTimeout field of the kubelet config. (kubernetes/kubernetes#131992, @lalitc375)

• Dynamic Resource Allocation: Graduated core functionality to general availability (GA). This newly stable feature uses the structured parameters flavor of DRA. (kubernetes/kubernetes#132706, @pohly) [SIG API Machinery, Apps, Auth, Autoscaling, Etcd, Node, Scheduling and Testing]

• Enabled kube-apiserver support for PodCertificateRequest and PodCertificate projected volumes (behind the PodCertificateRequest feature gate). (kubernetes/kubernetes#128010, @ahmedtd) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Storage and Testing]

• Extended resources backed by DRA feature allowed cluster operator to specify extendedResourceName in DeviceClass, and application operator to continue using extended resources in pod's requests to request for DRA devices matching the DeviceClass.

  NodeResourcesFit plugin scoring didn't work for extended resources backed by DRA. (kubernetes/kubernetes#130653, @yliaog) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• Extended the NodePorts scheduling plugin to consider hostPorts used by restartable init containers. (kubernetes/kubernetes#132040, @avrittrohwer) [SIG Scheduling and Testing]

• Fixed a 1.33 regression that causes a nil panic in kube-scheduler when aggregating resource requested across container's spec and status. (kubernetes/kubernetes#132895, @yue9944882) [SIG Node and Scheduling]

• Fixed prerelease lifecycle for PodCertificateRequest. (kubernetes/kubernetes#133350, @carlory)

• Introduced OpenAPI format support for k8s-short-name and k8s-long-name in CustomResourceDefinition schemas. (kubernetes/kubernetes#132504, @jpbetz) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling and Storage]

• Introduced the admissionregistration.k8s.io/v1beta1/MutatingAdmissionPolicy API type. To enable, enable the MutatingAdmissionPolicy feature gate (which was off by default) and set --runtime-config=admissionregistration.k8s.io/v1beta1=true on the kube-apiserver.
  Note that the default stored version remained alpha in 1.34, and whoever enabled beta during 1.34 needed to run a storage migration yourself to ensure you don't depend on alpha data in etcd. (kubernetes/kubernetes#132821, @cici37) [SIG API Machinery, Etcd and Testing]

• Kube-apiserver: Added support for disabling caching of authorization webhook decisions in the --authorization-config file. The new fields cacheAuthorizedRequests and cacheUnauthorizedRequests could be set to false to prevent caching for authorized or unauthorized requests. See the https://kubernetes.io/docs/reference/access-authn-authz/authorization/#using-configuration-file-for-authorization for more details. (kubernetes/kubernetes#129237, @rfranzke) [SIG API Machinery and Auth]

• Kube-apiserver: Promoted the StructuredAuthenticationConfiguration feature gate to GA. (kubernetes/kubernetes#131916, @aramase) [SIG API Machinery, Auth and Testing]

• Kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1. (kubernetes/kubernetes#131752, @aramase) [SIG API Machinery, Auth and Testing]

• Kube-log-runner: Added the -log-file-size parameter to rotate log output into a new file once it reached a certain size. Introduced -log-file-age to enable automatic removal of old output files, and -flush-interval to support periodic flushing. (kubernetes/kubernetes#127667, @zylxjtu) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Release, Scheduling, Storage, Testing and Windows]

• Kubectl: Graduated kuberc support to beta. A kuberc configuration file provided a mechanism for customizing kubectl behavior (distinct from kubeconfig, which configures cluster access across different clients). (kubernetes/kubernetes#131818, @soltysh) [SIG CLI and Testing]

• Promoted Job Pod Replacement Policy to general availability. The JobPodReplacementPolicy feature gate was locked to true and will be removed in a future Kubernetes release. (kubernetes/kubernetes#132173, @dejanzele) [SIG Apps and Testing]

• Promoted MutableCSINodeAllocatableCount to beta. (kubernetes/kubernetes#132429, @torredil)

• Promoted feature-gate VolumeAttributesClass to GA

  • Promoted API VolumeAttributesClass and VolumeAttributesClassList to storage.k8s.io/v1. (kubernetes/kubernetes#131549, @carlory) [SIG API Machinery, Apps, Auth, CLI, Etcd, Storage and Testing]

• Promoted the APIServerTracing feature gate to GA. The --tracing-config-file flag accepted TracingConfiguration in version apiserver.config.k8s.io/v1 (with no changes from apiserver.config.k8s.io/v1beta1). (kubernetes/kubernetes#132340, @dashpole) [SIG API Machinery and Testing]

• Promoted the AuthorizeWithSelectors and AuthorizeNodeWithSelectors feature gates to stable and locked on. (kubernetes/kubernetes#132656, @liggitt) [SIG API Machinery, Auth and Testing]

• Promoted the KubeletTracing feature gate to GA. (kubernetes/kubernetes#132341, @dashpole) [SIG Instrumentation and Node]

• Promoted the RelaxedEnvironmentVariableValidation feature gate to GA and locked it in the enabled state by default. (kubernetes/kubernetes#132054, @HirazawaUi) [SIG Apps, Architecture, Node and Testing]

• Removed an inaccurate statement about requiring ports when the Pod spec hostNetwork field was set. (kubernetes/kubernetes#130994, @BenTheElder) [SIG Network and Node]

• Removed deprecated gogo protocol definitions from k8s.io/kubelet/pkg/apis/pluginregistration in favor of google.golang.org/protobuf. (kubernetes/kubernetes#132773, @saschagrunert)

• Removed deprecated gogo protocol definitions from k8s.io/cri-api in favor of google.golang.org/protobuf. (kubernetes/kubernetes#128653, @saschagrunert) [SIG API Machinery, Auth, Instrumentation, Node and Testing]

• Replaced Boolean-pointer-helper functions with the k8s.io/utils/ptr implementations. (kubernetes/kubernetes#132794, @PatrickLaabs) [SIG API Machinery, Auth, CLI, Node and Testing]

• Replaced boolPtrFn helper functions with the "k8s.io/utils/ptr" implementation. (kubernetes/kubernetes#132907, @PatrickLaabs)

• Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the apiextensions-apiserver apiextensions. (kubernetes/kubernetes#132723, @PatrickLaabs)

• Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the apiserver (1/2). (kubernetes/kubernetes#132751, @PatrickLaabs) [SIG API Machinery and Auth]

• Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the component-base. (kubernetes/kubernetes#132754, @PatrickLaabs) [SIG API Machinery, Architecture, Instrumentation and Scheduling]

• Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the kube-aggregator apiregistration. (kubernetes/kubernetes#132701, @PatrickLaabs)

• Simplied validation error message for invalid fields by removing redundant field name. (kubernetes/kubernetes#132513, @xiaoweim) [SIG API Machinery, Apps, Auth, Node and Scheduling]

• Simplied validation error message for required fields by removing redundant messages. (kubernetes/kubernetes#132472, @xiaoweim) [SIG API Machinery, Apps, Architecture, Auth, Cloud Provider, Network, Node and Storage]

• The KubeletServiceAccountTokenForCredentialProviders feature was beta and enabled by default. (kubernetes/kubernetes#133017, @aramase) [SIG Auth and Node]

• The conditionType is "oneof" approved/denied check of CertificateSigningRequest's .status.conditions field was migrated to declarative validation.
  If the DeclarativeValidation feature gate was enabled, mismatches with existing validation are reported via metrics.
  If the DeclarativeValidationTakeover feature gate was enabled, declarative validation was the primary source of errors for migrated fields. (kubernetes/kubernetes#133013, @aaron-prindle) [SIG API Machinery and Auth]

• The fallback behavior of the Downward API's resourceFieldRef field was updated to account for pod-level resources: if container-level limits were not set, pod-level limits were now used before falling back to node allocatable resources. (kubernetes/kubernetes#132605, @toVersus) [SIG Node, Scheduling and Testing]

• The validation of replicas field in the ReplicationController /scale subresource has been migrated to declarative validation.
  If the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics.
  If the DeclarativeValidationTakeover feature gate is enabled, declarative validation is the primary source of errors for migrated fields. (kubernetes/kubernetes#131664, @jpbetz) [SIG API Machinery and Apps]

• The validation-gen code generator generated validation code that supported validation ratcheting. (kubernetes/kubernetes#132236, @yongruilin) [SIG API Machinery, Apps, Auth and Node]

• Updated IsDNS1123SubdomainWithUnderscore so that, when it returned an error, it also returned the correct regex information (dns1123SubdomainFmtWithUnderscore). (kubernetes/kubernetes#132034, @ChosenFoam)

• Updated etcd version to v3.6.0. (kubernetes/kubernetes#131501, @joshjms) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing]

• Updated the v1 credential provider configuration to include the tokenAttributes.cacheType field. This field is required and must be set to either ServiceAccount or Token when configuring a provider that uses a service account to fetch registry credentials. (kubernetes/kubernetes#132617, @aramase) [SIG Auth, Node and Testing]

• Zero-value metadata.creationTimestamp values are now omitted and no longer serialize an explicit null in JSON, YAML, and CBOR output (kubernetes/kubernetes#130989, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• AppArmor profiles specified in the Pod or container SecurityContext were no longer copied to deprecated AppArmor annotations (prefix container.apparmor.security.beta.kubernetes.io/). Anything that inspected the deprecated annotations must be migrated to use the SecurityContext fields instead. (kubernetes/kubernetes#131989, @tallclair)

• MultiCIDRServiceAllocator was locked and enabled by default, DisableAllocatorDualWrite was enabled by default. (kubernetes/kubernetes#131318, @aojea) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Network and Testing]

• Add a new FileKeyRef field to containers, allowing them to load variables from files by setting this field.

  Introduce the EnvFiles feature gate to govern activation of this functionality. (kubernetes/kubernetes#132626, @HirazawaUi) [SIG API Machinery, Apps, Node and Testing]

• Add driver-owned fields in ResourceSlice to mark whether the device is shareable among multiple resource claims (or requests) and to specify how each capacity can be shared between different requests.

  • Add user-owned fields in ResourceClaim to specify resource requirements against each device capacity.
  • Add scheduler-owned field in ResourceClaim.Status to specify how much device capacity is reserved for a specific request.
  • Add an additional identifier to ResourceClaim.Status for the device supports multiple allocations.
  • Add a new constraint type to enforce uniqueness of specified attributes across all allocated devices. (kubernetes/kubernetes#132522, @sunya-ch) [SIG API Machinery, Apps, Architecture, CLI, Cluster Lifecycle, Network, Node, Release, Scheduling and Testing]

• Add new optional APIs in ResouceSlice.Basic and ResourceClaim.Status.AllocatedDeviceStatus. (kubernetes/kubernetes#130160, @KobayashiD27) [SIG API Machinery, Apps, Architecture, Node, Release, Scheduling and Testing]

• Added a mechanism for configurable container restarts: container level restart rules. This is an alpha feature behind the ContainerRestartRules feature gate. (kubernetes/kubernetes#132642, @yuanwang04) [SIG API Machinery, Apps, Node and Testing]

• Added detailed event for in-place pod vertical scaling completed, improving cluster management and debugging (kubernetes/kubernetes#130387, @shiya0705) [SIG API Machinery, Apps, Autoscaling, Node, Scheduling and Testing]

• Added validation to reject Pods using the PodLevelResources feature on Windows OS due to lack of support. The API server rejects Pods with Pod-level resources and a Pod.spec.os.name targeting Windows. Kubelet on nodes running Windows also rejects Pods with Pod-level resources at admission phase. (kubernetes/kubernetes#133046, @toVersus) [SIG Apps and Node]

• Adds warnings when creating headless service with set loadBalancerIP,externalIPs and/or SessionAffinity (kubernetes/kubernetes#132214, @Peac36) [SIG Network]

• Allow pvc.spec.VolumeAttributesClassName to go from non-nil to nil (kubernetes/kubernetes#132106, @AndrewSirenko) [SIG Apps]

• Allows setting the hostnameOverride field in PodSpec to specify any RFC 1123 DNS subdomain as the pod's hostname. The HostnameOverride feature gate has been introduced to control enablement of this functionality. (kubernetes/kubernetes#132558, @HirazawaUi) [SIG API Machinery, Apps, Network, Node and Testing]

• AppArmor profiles specified in the pod or container SecurityContext are no longer copied to deprecated AppArmor annotations (prefix container.apparmor.security.beta.kubernetes.io/). Anything that inspects the deprecated annotations must be migrated to use the SecurityContext fields instead. (kubernetes/kubernetes#131989, @tallclair) [SIG Node]

• Changes underlying logic to propagate Pod level hugepage cgroup to containers when they do not specify hugepage resources.

  • Adds validation to enforce the hugepage aggregated container limits to be smaller or equal to pod-level limits. This was already enforced with the defaulted requests from the specified limits, however it did not make it clear about both hugepage requests and limits. (kubernetes/kubernetes#131089, @KevinTMtz) [SIG Apps, Node and Testing]

• DRA: the scheduler plugin now prevents abnormal filter runtimes by timing out after 10 seconds. This is configurable via the plugin configuration's FilterTimeout. Setting it to zero disables the timeout and restores the behavior of Kubernetes <= 1.33. (kubernetes/kubernetes#132033, @pohly) [SIG Node, Scheduling and Testing]

• DRA: when the prioritized list feature is used in a request and the resulting number of allocated devices exceeds the number of allowed devices per claim, the scheduler aborts the attempt to allocate devices early. Previously it tried to many different combinations, which can take a long time. (kubernetes/kubernetes#130593, @mortent) [SIG Apps, Node, Scheduling and Testing]

• Dynamic Resource Allocation: graduated core functionality to general availability (GA). This newly stable feature uses the structured parameters flavor of DRA. (kubernetes/kubernetes#132706, @pohly) [SIG API Machinery, Apps, Auth, Autoscaling, Etcd, Node, Scheduling and Testing]

• Enable kube-apiserver support for PodCertificateRequest and PodCertificate projected volumes (behind the PodCertificateRequest feature gate). (kubernetes/kubernetes#128010, @ahmedtd) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Storage and Testing]

• Extended resources backed by DRA feature allows cluster operator to specify extendedResourceName in DeviceClass, and application operator to continue using extended resources in pod's requests to request for DRA devices matching the DeviceClass.

  NodeResourcesFit plugin scoring won't work for extended resources backed by DRA (kubernetes/kubernetes#130653, @yliaog) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• Fix prerelease lifecycle for PodCertificateRequest (kubernetes/kubernetes#133350, @carlory) [SIG Auth]

• Fixes a 1.33 regression that can cause a nil panic in kube-scheduler when aggregating resource requests across container's spec and status. (kubernetes/kubernetes#132895, @yue9944882) [SIG Node and Scheduling]

• Introduced the admissionregistration.k8s.io/v1beta1/MutatingAdmissionPolicy API type.
  To enable, enable the MutatingAdmissionPolicy feature gate (which is off by default) and set --runtime-config=admissionregistration.k8s.io/v1beta1=true on the kube-apiserver.
  Note that the default stored version remains alpha in 1.34 and whoever enabled beta during 1.34 needs to run a storage migration yourself to ensure you don't depend on alpha data in etcd. (kubernetes/kubernetes#132821, @cici37) [SIG API Machinery, Etcd and Testing]

• No, changes underlying logic for Eviction Manager helper functions (kubernetes/kubernetes#132277, @KevinTMtz) [SIG Node, Scheduling and Testing]

• Promote MutableCSINodeAllocatableCount to Beta. (kubernetes/kubernetes#132429, @torredil) [SIG Storage]

• Promoted feature-gate VolumeAttributesClass to GA

  • Promoted API VolumeAttributesClass and VolumeAttributesClassList to storage.k8s.io/v1. (kubernetes/kubernetes#131549, @carlory) [SIG API Machinery, Apps, Auth, CLI, Etcd, Storage and Testing]

• Promoted the APIServerTracing feature gate to GA. The --tracing-config-file flag now accepts TracingConfiguration in version apiserver.config.k8s.io/v1 (with no changes from apiserver.config.k8s.io/v1beta1). (kubernetes/kubernetes#132340, @dashpole) [SIG API Machinery and Testing]

• Removed deprecated gogo protocol definitions from k8s.io/kubelet/pkg/apis/pluginregistration in favor of google.golang.org/protobuf. (kubernetes/kubernetes#132773, @saschagrunert) [SIG Node]

• The Kubelet can now monitor the health of devices allocated via Dynamic Resource Allocation (DRA) and report it in the pod.status.containerStatuses.allocatedResourcesStatus field. This requires the DRA plugin to implement the new v1alpha1 NodeHealth gRPC service. This feature is controlled by the ResourceHealthStatus feature gate. (kubernetes/kubernetes#130606, @Jpsassine) [SIG Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Network, Node, Release, Scheduling, Storage and Testing]

• The KubeletServiceAccountTokenForCredentialProviders feature is now beta and enabled by default. (kubernetes/kubernetes#133017, @aramase) [SIG Auth and Node]

• The conditionType is "oneof" approved/denied check of CertificateSigningRequest's .status.conditions field has been migrated to declarative validation.
  If the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics.
  If the DeclarativeValidationTakeover feature gate is enabled, declarative validation is the primary source of errors for migrated fields. (kubernetes/kubernetes#133013, @aaron-prindle) [SIG API Machinery and Auth]

• The fallback behavior of the Downward API's resourceFieldRef field has been updated to account for pod-level resources: if container-level limits are not set, pod-level limits are now used before falling back to node allocatable resources. (kubernetes/kubernetes#132605, @toVersus) [SIG Node, Scheduling and Testing]

• The kubelet's image pull credential tracking now supports service account-based verification. When an image is pulled using service account credentials via external credential providers, subsequent pods using the same service account (UID, name, and namespace) can access the cached image without re-authentication for the lifetime of that service account. (kubernetes/kubernetes#132771, @aramase) [SIG Auth, Node and Testing]

• Added tokenAttributes.cacheType field to v1 credential provider config. This field is required to be set to either ServiceAccount or Token when configuring a provider that uses service account to fetch registry credentials. (kubernetes/kubernetes#132617, @aramase) [SIG Auth, Node and Testing]

• JWT authenticators specified via the AuthenticationConfiguration.jwt array can now optionally specify either the controlplane or cluster egress selector by setting the issuer.egressSelectorType field. When unset, the prior behavior of using no egress selector is retained. The StructuredAuthenticationConfigurationEgressSelector beta feature (default on) must be enabled to use this functionality. (kubernetes/kubernetes#132768, @enj) [SIG API Machinery, Auth and Testing]

• Promoted the KubeletTracing feature gate to GA. (kubernetes/kubernetes#132341, @dashpole) [SIG Instrumentation and Node]

• Replaces boolPtrFn helper functions with the "k8s.io/utils/ptr" implementation. (kubernetes/kubernetes#132907, @PatrickLaabs) [SIG Architecture]

• Simplied validation error message for invalid fields by removing redundant field name. (kubernetes/kubernetes#132513, @xiaoweim) [SIG API Machinery, Apps, Auth, Node and Scheduling]

• The AuthorizeWithSelectors and AuthorizeNodeWithSelectors feature gates are promoted to stable and locked on. (kubernetes/kubernetes#132656, @liggitt) [SIG API Machinery, Auth and Testing]

• DRA: the v1alpha4 kubelet gRPC API (added in 1.31, superseded in 1.32) is no longer supported. DRA drivers using the helper package from Kubernetes >= 1.32 use the v1beta1 API and continue to be supported. (kubernetes/kubernetes#132574, @pohly) [SIG Node]

• Deprecate StreamingConnectionIdleTimeout field of the kubelet config. (kubernetes/kubernetes#131992, @lalitc375) [SIG Node]

• Removed deprecated gogo protocol definitions from k8s.io/cri-api in favor of google.golang.org/protobuf. (kubernetes/kubernetes#128653, @saschagrunert) [SIG API Machinery, Auth, Instrumentation, Node and Testing]

• Replaced deprecated package 'k8s.io/utils/pointer' with 'k8s.io/utils/ptr' for the apiextensions-apiserver apiextensions. (kubernetes/kubernetes#132723, @PatrickLaabs) [SIG API Machinery]

• Replaced deprecated package 'k8s.io/utils/pointer' with 'k8s.io/utils/ptr' for the component-base. (kubernetes/kubernetes#132754, @PatrickLaabs) [SIG API Machinery, Architecture, Instrumentation and Scheduling]

• Replaced deprecated package 'k8s.io/utils/pointer' with 'k8s.io/utils/ptr' for the kube-aggregator apiregistration. (kubernetes/kubernetes#132701, @PatrickLaabs) [SIG API Machinery]

• Replaces Boolean-pointer-helper functions with the "k8s.io/utils/ptr" implementations. (kubernetes/kubernetes#132794, @PatrickLaabs) [SIG API Machinery, Auth, CLI, Node and Testing]

• Replaces deprecated package 'k8s.io/utils/pointer' with 'k8s.io/utils/ptr' for the apiserver (1/2). (kubernetes/kubernetes#132751, @PatrickLaabs) [SIG API Machinery and Auth]

• Simplied validation error message for required fields by removing redundant messages. (kubernetes/kubernetes#132472, @xiaoweim) [SIG API Machinery, Apps, Architecture, Auth, Cloud Provider, Network, Node and Storage]

• Add a runtime.ApplyConfiguration interface that is implemented by all generated applyconfigs (kubernetes/kubernetes#132194, @alvaroaleman) [SIG API Machinery and Instrumentation]

• Added omitempty and opt tag to the API v1beta2 AdminAccess type in the DeviceRequestAllocationResult struct. (kubernetes/kubernetes#132338, @PatrickLaabs) [SIG Auth]

• Introduces OpenAPI format support for k8s-short-name and k8s-long-name. (kubernetes/kubernetes#132504, @jpbetz) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling and Storage]

• Promoted Job Pod Replacement Policy to general availability. The JobPodReplacementPolicy feature gate is now locked to true, and will be removed in a future release of Kubernetes. (kubernetes/kubernetes#132173, @dejanzele) [SIG Apps and Testing]

• This PR corrects that documentation, making it clear to users that podSelector is optional and describes its default behavior. (kubernetes/kubernetes#131354, @tomoish) [SIG Network]

• Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

  (kubernetes/kubernetes#131996, @ritazh) [SIG Node and Testing]

• DRA API: resource.k8s.io/v1alpha3 now only contains DeviceTaintRule. All other types got removed because they became obsolete when introducing the v1beta1 API in 1.32.
  before updating a cluster where resourceclaims, resourceclaimtemplates, deviceclasses, or resourceslices might have been stored using Kubernetes < 1.32, delete all of those resources before updating and recreate them as needed while running Kubernetes >= 1.32. (kubernetes/kubernetes#132000, @pohly) [SIG Etcd, Node, Scheduling and Testing]

• Extends the nodeports scheduling plugin to consider hostPorts used by restartable init containers. (kubernetes/kubernetes#132040, @avrittrohwer) [SIG Scheduling and Testing]

• Kube-apiserver: Caching of authorization webhook decisions for authorized and unauthorized requests can now be disabled in the --authorization-config file by setting the new fields cacheAuthorizedRequests or cacheUnauthorizedRequests to false explicitly. See https://kubernetes.io/docs/reference/access-authn-authz/authorization/#using-configuration-file-for-authorization for more details. (kubernetes/kubernetes#129237, @rfranzke) [SIG API Machinery and Auth]

• Kube-apiserver: Promoted the StructuredAuthenticationConfiguration feature gate to GA. (kubernetes/kubernetes#131916, @aramase) [SIG API Machinery, Auth and Testing]

• Kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1. (kubernetes/kubernetes#131752, @aramase) [SIG API Machinery, Auth and Testing]

• Kube-log-runner: rotating log output into a new file when reaching a certain file size can be requested via the new -log-file-size parameter. -log-file-age enables automatical removal of old output files. Periodic flushing can be requested through -flush-interval. (kubernetes/kubernetes#127667, @zylxjtu) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Release, Scheduling, Storage, Testing and Windows]

• Kubectl: graduated kuberc support to beta. A kuberc configuration file provides a mechanism for customizing kubectl behavior (separate from kubeconfig, which configured cluster access across different clients). (kubernetes/kubernetes#131818, @soltysh) [SIG CLI and Testing]

• Promote the RelaxedEnvironmentVariableValidation feature gate to GA and lock it in the default enabled state. (kubernetes/kubernetes#132054, @HirazawaUi) [SIG Apps, Architecture, Node and Testing]

• Remove inaccurate statement about requiring ports from pod spec hostNetwork field (kubernetes/kubernetes#130994, @BenTheElder) [SIG Network and Node]

• TBD (kubernetes/kubernetes#131318, @aojea) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Network and Testing]

• The validation of replicas field in the ReplicationController /scale subresource has been migrated to declarative validation.
  If the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics.
  If the DeclarativeValidationTakeover feature gate is enabled, declarative validation is the primary source of errors for migrated fields. (kubernetes/kubernetes#131664, @jpbetz) [SIG API Machinery and Apps]

• The validation-gen code generator generates validation code that supports validation ratcheting. (kubernetes/kubernetes#132236, @yongruilin) [SIG API Machinery, Apps, Auth and Node]

• Update etcd version to v3.6.0 (kubernetes/kubernetes#131501, @joshjms) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing]

• When the IsDNS1123SubdomainWithUnderscore function returns an error, it will return the correct regex information dns1123SubdomainFmtWithUnderscore. (kubernetes/kubernetes#132034, @ChosenFoam) [SIG Network]

• Zero-value metadata.creationTimestamp values are now omitted and no longer serialize an explicit null in JSON, YAML, and CBOR output (kubernetes/kubernetes#130989, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scheduling, Storage and Testing]

v33.0.0+snapshot

Kubernetes API Version: v1.33.1

API Change Copy Section

• A new alpha feature gate, MutableCSINodeAllocatableCount, has been introduced.

  When this feature gate is enabled, the CSINode.Spec.Drivers[*].Allocatable.Count field becomes mutable, and a new field, NodeAllocatableUpdatePeriodSeconds, is available in the CSIDriver object. This allows periodic updates to a node's reported allocatable volume capacity, preventing stateful pods from becoming stuck due to outdated information that kube-scheduler relies on. (kubernetes/kubernetes#130007, @torredil) [SIG Apps, Node, Scheduling and Storage]

• Added feature gate DRAPartitionableDevices, when enabled, Dynamic Resource Allocation support partitionable devices allocation. (kubernetes/kubernetes#130764, @cici37) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• Added DRA support for a "one-of" prioritized list of selection criteria to satisfy a device request in a resource claim. (kubernetes/kubernetes#128586, @mortent) [SIG API Machinery, Apps, Etcd, Node, Scheduling and Testing]

• Added a /flagz endpoint for kubelet endpoint (kubernetes/kubernetes#128857, @zhifei92) [SIG Architecture, Instrumentation and Node]

• Added a new tolerance field to HorizontalPodAutoscaler, overriding the cluster-wide default. Enabled via the HPAConfigurableTolerance alpha feature gate. (kubernetes/kubernetes#130797, @jm-franc) [SIG API Machinery, Apps, Autoscaling, Etcd, Node, Scheduling and Testing]

• Added support for configuring custom stop signals with a new StopSignal container lifecycle (kubernetes/kubernetes#130556, @sreeram-venkitesh) [SIG API Machinery, Apps, Node and Testing]

• Added support for in-place vertical scaling of Pods with sidecars (containers defined within initContainers where the restartPolicy is set to Always). (kubernetes/kubernetes#128367, @vivzbansal) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]

• CPUManager Policy Options support is GA (kubernetes/kubernetes#130535, @ffromani) [SIG API Machinery, Node and Testing]

• Changed the Pod API to support hugepage resources at spec level for pod-level resources. (kubernetes/kubernetes#130577, @KevinTMtz) [SIG Apps, CLI, Node, Scheduling, Storage and Testing]

• DRA API: The maximum number of pods that can use the same ResourceClaim is now 256 instead of 32. Downgrading a cluster where this relaxed limit is in use to Kubernetes 1.32.0 is not supported, as version 1.32.0 would refuse to update ResourceClaims with more than 32 entries in the status.reservedFor field. (kubernetes/kubernetes#129543, @pohly) [SIG API Machinery, Node and Testing]

• DRA: CEL expressions using attribute strings exceeded the cost limit because their cost estimation was incomplete. (kubernetes/kubernetes#129661, @pohly) [SIG Node]

• DRA: Device taints enable DRA drivers or admins to mark device as unusable, which prevents allocating them. Pods may also get evicted at runtime if a device becomes unusable, depending on the severity of the taint and whether the claim tolerates the taint. (kubernetes/kubernetes#130447, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]

• DRA: Starting Kubernetes 1.33, only users with access to an admin namespace with the kubernetes.io/dra-admin-access label are authorized to create ResourceClaim or ResourceClaimTemplate objects with the adminAccess field in this admin namespace if they want to and only they can reference these ResourceClaims or ResourceClaimTemplates in their pod or deployment specs. (kubernetes/kubernetes#130225, @ritazh) [SIG API Machinery, Apps, Auth, Node and Testing]

• DRA: when asking for "All" devices on a node, Kubernetes <= 1.32 proceeded to schedule pods onto nodes with no devices by not allocating any devices for those pods. Kubernetes 1.33 changes that to only picking nodes which have at least one device. Users who want the "proceed with scheduling also without devices" semantic can use the upcoming prioritized list feature with one sub-request for "all" devices and a second alternative with "count: 0". (kubernetes/kubernetes#129560, @bart0sh) [SIG API Machinery and Node]

• Expanded the on-disk kubelet credential provider configuration to allow an optional tokenAttribute field to be configured. When it is set, the kubelet will provision a token with the given audience bound to the current pod and its service account. This KSA token along with required annotations on the KSA defined in configuration will be sent to the credential provider plugin via its standard input (along with the image information that is already sent today). The KSA annotations to be sent are configurable in the kubelet credential provider configuration. (kubernetes/kubernetes#128372, @aramase) [SIG API Machinery, Auth, Node and Testing]

• Fixed the example validation rule in godoc:

  When configuring a JWT authenticator:

  If username.expression uses 'claims.email', then 'claims.email_verified' must be used in
  username.expression or extra[].valueExpression or claimValidationRules[].expression.
  An example claim validation rule expression that matches the validation automatically
  applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true) == true'.
  By explicitly comparing the value to true, we let type-checking see the result will be a boolean,
  and to make sure a non-boolean email_verified claim will be caught at runtime. (kubernetes/kubernetes#130875, @aramase) [SIG Auth and Release]

• For the InPlacePodVerticalScaling feature, the API server will no longer set the resize status to Proposed upon receiving a resize request. (kubernetes/kubernetes#130574, @natasha41575) [SIG Apps, Node and Testing]

• Graduate the MatchLabelKeys (MismatchLabelKeys) feature in PodAffinity (PodAntiAffinity) to GA (kubernetes/kubernetes#130463, @sanposhiho) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Graduated image volume sources to beta:

  • Allowed subPath/subPathExpr for image volumes
  • Added kubelet metrics kubelet_image_volume_requested_total, kubelet_image_volume_mounted_succeed_total and kubelet_image_volume_mounted_errors_total (kubernetes/kubernetes#130135, @saschagrunert) [SIG API Machinery, Apps, Node and Testing]

• Implemented a new status field, .status.terminatingReplicas, for Deployments and ReplicaSets to track terminating pods. The new field is present when the DeploymentPodReplacementPolicy feature gate is enabled. (kubernetes/kubernetes#128546, @atiratree) [SIG API Machinery, Apps and Testing]

• Implemented validation for NodeSelectorRequirement values in Kubernetes when creating pods. (kubernetes/kubernetes#128212, @AxeZhan) [SIG Apps and Scheduling]

• Improved how the API server responds to list requests where the response format negotiates to Protobuf. List responses in Protobuf are marshalled one element at the time, drastically reducing memory needed to serve large collections. Streaming list responses can be disabled via the StreamingCollectionEncodingToProtobuf feature gate. (kubernetes/kubernetes#129407, @serathius) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Network, Node, Release, Scheduling, Storage and Testing]

• InPlacePodVerticalScaling: Memory limits cannot be decreased unless the memory resize restart policy is set to RestartContainer. Container resizePolicy is no longer mutable. (kubernetes/kubernetes#130183, @tallclair) [SIG Apps and Node]

• Introduced API type coordination.k8s.io/v1beta1/LeaseCandidate
CoordinatedLeaderElection feature moves to Beta (kubernetes/kubernetes#130751, @Jefftree) [SIG API Machinery, Etcd and Testing]

• Introduced API type coordination.k8s.io/v1beta1/LeaseCandidate (kubernetes/kubernetes#130291, @Jefftree) [SIG API Machinery, Etcd and Testing]

• It introduces a new scope name VolumeAttributesClass.

  It matches all PVC objects that have the volume attributes class mentioned.

  If you want to limit the count of PVCs that have a specific volume attributes class. In that case, you can create a quota object with the scope name VolumeAttributesClass and a matchExpressions that match the volume attributes class. (kubernetes/kubernetes#124360, @carlory) [SIG API Machinery, Apps and Testing]

• KEP-3857: Recursive Read-only (RRO) mounts: promote to GA (kubernetes/kubernetes#130116, @AkihiroSuda) [SIG Apps, Node and Testing]

• kubectl: Added alpha support for customizing kubectl behavior using preferences from a kuberc file, separate from kubeconfig. (kubernetes/kubernetes#125230, @ardaguclu) [SIG API Machinery, CLI and Testing]

• kubelet: added KubeletConfiguration.subidsPerPod. (kubernetes/kubernetes#130028, @AkihiroSuda) [SIG API Machinery and Node]

• Kubernetes components that accepted X.509 client certificate authentication now read the user UID from a certificate subject name RDN with object ID 1.3.6.1.4.1.57683.2. An RDN with this object ID had to contain a string value and appear no more than once in the certificate subject. Reading the user UID from this RDN could be disabled by setting the beta feature gate AllowParsingUserUIDFromCertAuth to false(until the feature gate graduated to GA). (kubernetes/kubernetes#127897, @modulitos) [SIG API Machinery, Auth and Testing]

• MergeDefaultEvictionSettings indicates that defaults for the evictionHard, evictionSoft, evictionSoftGracePeriod, and evictionMinimumReclaim fields should be merged into values specified for those fields in this configuration. Signals specified in this configuration take precedence. Signals not specified in this configuration inherit their defaults. (kubernetes/kubernetes#127577, @vaibhav2107) [SIG API Machinery and Node]

• New configuration is introduced to the kubelet that allows it to track container images and the list of authentication information that leads to their successful pulls. This data is persisted across reboots of the host and restarts of the kubelet.

  The kubelet ensures any image requiring credential verification is always pulled if authentication information from an image pull is not yet present, thus enforcing authentication / re-authentication. This means an image pull might be attempted even in cases where a pod requests the IfNotPresent image pull policy, and might lead to the pod not starting if its pull policy is Never and is unable to present authentication information that led to a previous successful pull of the image it is requesting. (kubernetes/kubernetes#128152, @stlaz) [SIG API Machinery, Architecture, Auth, Node and Testing]

• Promoted JobSuccessPolicy E2E to Conformance (kubernetes/kubernetes#130658, @tenzen-y) [SIG API Machinery, Apps, Architecture and Testing]

• Promoted NodeInclusionPolicyInPodTopologySpread to Stable in v1.33 (kubernetes/kubernetes#130920, @kerthcet) [SIG Apps, Node, Scheduling and Testing]

• Promoted the JobSuccessPolicy to Stable. (kubernetes/kubernetes#130536, @tenzen-y) [SIG API Machinery, Apps, Architecture and Testing]

• Promoted the Job's JobBackoffLimitPerIndex feature-gate to stable. (kubernetes/kubernetes#130061, @mimowo) [SIG API Machinery, Apps, Architecture and Testing]

• Promoted the feature gate AnyVolumeDataSource to GA. (kubernetes/kubernetes#129770, @sunnylovestiramisu) [SIG Apps, Storage and Testing]

• Removed general available feature gate CPUManager. (kubernetes/kubernetes#129296, @carlory) [SIG API Machinery, Node and Testing]

• Removed general available feature-gate PDBUnhealthyPodEvictionPolicy. (kubernetes/kubernetes#129500, @carlory) [SIG API Machinery, Apps and Auth]

• Start reporting swap capacity as part of node.status.nodeSystemInfo. (kubernetes/kubernetes#129954, @iholder101) [SIG API Machinery, Apps and Node]

• Graduated the MultiCIDRServiceAllocator feature gate to stable, and the DisableAllocatorDualWrite feature gate to beta (disabled by default).
  Action required for Kubernetes cluster administrators and for distributions that manage the cluster Service CIDR.
  Kubernetes now allows users to define the cluster Service CIDR via an API object: ServiceCIDR.
  Distributions or administrators of Kubernetes may want to control that new Service CIDRs added to the cluster do not overlap with other networks on the cluster, that only belong to a specific range of IPs. Administrators may also prefer to retain the existing behavior of only having one ServiceCIDR per cluster. You can use ValidatingAdmissionPolicy to achieve this. (kubernetes/kubernetes#128971, @aojea) [SIG Apps, Architecture, Auth, CLI, Etcd, Network, Release and Testing]

• The ClusterTrustBundle API is moving to v1beta1.
  In order for the ClusterTrustBundleProjection feature to work on the kubelet side, the ClusterTrustBundle API must be available at v1beta1 version and the ClusterTrustBundleProjection feature gate must be enabled. If the API becomes later after kubelet started running, restart the kubelet to enable the feature. (kubernetes/kubernetes#128499, @stlaz) [SIG API Machinery, Apps, Auth, Etcd, Node, Storage and Testing]

• The Service trafficDistribution field, including the PreferClose option, has graduated
  to GA. Services that do not have the field configured will continue to operate
  with their existing behavior. Refer to the documentation
  https://kubernetes.io/docs/concepts/services-networking/service/#traffic-distribution
  for more details. (kubernetes/kubernetes#130673, @gauravkghildiyal) [SIG Apps, Network and Testing]

• The feature gate InPlacePodVerticalScalingAllocatedStatus is deprecated and no longer used. The AllocatedResources field in ContainerStatus is now guarded by the InPlacePodVerticalScaling feature gate. (kubernetes/kubernetes#130880, @tallclair) [SIG CLI, Node and Scheduling]

• The kube-controller-manager will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (kubernetes/kubernetes#130650, @natasha41575) [SIG API Machinery, Apps, Node, Scheduling, Storage, Testing and Windows]

• The kube-scheduler will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (kubernetes/kubernetes#130649, @natasha41575) [SIG Node, Scheduling and Testing]

• The kubelet will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (kubernetes/kubernetes#130573, @natasha41575) [SIG Apps, Node, Scheduling, Storage, Testing and Windows]

• The minimum value validation of ReplicationController's replicas and minReadySeconds fields have been migrated to declarative validation. The requiredness of both fields is also declaratively validated.
  If the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics.
  If the DeclarativeValidationTakeover feature gate is enabled, declarative validation is the primary source of errors for migrated fields. (kubernetes/kubernetes#130725, @jpbetz) [SIG API Machinery, Apps, Architecture, CLI, Cluster Lifecycle, Instrumentation, Network, Node and Storage]

• The resource.k8s.io/v1beta1 API is deprecated and will be removed in 1.36. Use v1beta2 instead. (kubernetes/kubernetes#129970, @mortent) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• Validation now requires new StatefulSets with a .spec.serviceName field value to pass DNS1123 validation. Previously created StatefulSets with an invalid .spec.serviceName field value could not create any pods, and should be deleted.

  • Published OpenAPI for the StatefulSet schema is corrected to indicate the .spec.serviceName is optional. (kubernetes/kubernetes#130233, @soltysh) [SIG API Machinery, Apps and Testing]

• When the PreferSameTrafficDistribution feature gate is enabled, a new trafficDistribution value PreferSameNode is available, which attempts to always route Service connections to an endpoint on the same node as the client. Additionally, PreferSameZone is introduced as an alias for PreferClose. (kubernetes/kubernetes#130844, @danwinship) [SIG API Machinery, Apps, Network and Windows]

• When the PodObservedGenerationTracking feature gate was set, the kubelet populated status.observedGeneration to reflect the latest metadata.generation it observed for the pod. (kubernetes/kubernetes#130352, @natasha41575) [SIG API Machinery, Apps, CLI, Node, Release, Scheduling, Storage, Testing and Windows]

• When the StrictIPCIDRValidation feature gate is enabled, Kubernetes will be
  slightly stricter about what values will be accepted as IP addresses and network
  address ranges (“CIDR blocks”).

  In particular, octets within IPv4 addresses are not allowed to have any leading
  0s, and IPv4-mapped IPv6 values (e.g. ::ffff:192.168.0.1) are forbidden.
  These sorts of values can potentially cause security problems when different
  components interpret the same string as referring to different IP addresses
  (as in CVE-2021-29923).

  This tightening applies only to fields in built-in API kinds, and not to
  custom resource kinds, values in Kubernetes configuration files, or
  command-line arguments.

  (When the feature gate is disabled, creating an object with such an invalid
  IP or CIDR value will result in a warning from the API server about the fact
  that it will be rejected in the future.) (kubernetes/kubernetes#122550, #128786, @danwinship) [SIG API Machinery, Apps, Network, Node, Scheduling and Testing]

• apidiscovery.k8s.io/v2beta1 API group is disabled by default (kubernetes/kubernetes#130347, @Jefftree) [SIG API Machinery and Testing]

• kubectl apply now coerces null values for labels and annotations in manifests to empty string values,
  consistent with typed JSON metadata decoding, rather than dropping all labels and annotations (kubernetes/kubernetes#129257, @liggitt) [SIG API Machinery]

• A new alpha feature gate, MutableCSINodeAllocatableCount, has been introduced.

  When this feature gate is enabled, the CSINode.Spec.Drivers[*].Allocatable.Count field becomes mutable, and a new field, NodeAllocatableUpdatePeriodSeconds, is available in the CSIDriver object. This allows periodic updates to a node's reported allocatable volume capacity, preventing stateful pods from becoming stuck due to outdated information that kube-scheduler relies on. (kubernetes/kubernetes#130007, @torredil) [SIG Apps, Node, Scheduling and Storage]

• Add feature gate DRAPartitionableDevices, when enabled, Dynamic Resource Allocation support partitionable devices allocation. (kubernetes/kubernetes#130764, @cici37) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• Added a /flagz endpoint for kubelet endpoint (kubernetes/kubernetes#128857, @zhifei92) [SIG Architecture, Instrumentation and Node]

• Added a new 'tolerance' field to HorizontalPodAutoscaler, overriding the cluster-wide default. Enabled via the HPAConfigurableTolerance alpha feature gate. (kubernetes/kubernetes#130797, @jm-franc) [SIG API Machinery, Apps, Autoscaling, Etcd, Node, Scheduling and Testing]

• Added support for configuring custom stop signals with a new StopSignal container lifecycle (kubernetes/kubernetes#130556, @sreeram-venkitesh) [SIG API Machinery, Apps, Node and Testing]

• CPUManager Policy Options support is GA (kubernetes/kubernetes#130535, @ffromani) [SIG API Machinery, Node and Testing]

• Changed the Pod API to support hugepage resources at spec level for pod-level resources. (kubernetes/kubernetes#130577, @KevinTMtz) [SIG Apps, CLI, Node, Scheduling, Storage and Testing]

• DRA: Device taints enable DRA drivers or admins to mark device as unusable, which prevents allocating them. Pods may also get evicted at runtime if a device becomes unusable, depending on the severity of the taint and whether the claim tolerates the taint. (kubernetes/kubernetes#130447, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]

• DRA: Starting Kubernetes 1.33, only users with access to an admin namespace with the kubernetes.io/dra-admin-access label are authorized to create ResourceClaim or ResourceClaimTemplate objects with the adminAccess field in this admin namespace if they want to and only they can reference these ResourceClaims or ResourceClaimTemplates in their pod or deployment specs. (kubernetes/kubernetes#130225, @ritazh) [SIG API Machinery, Apps, Auth, Node and Testing]

• Expanded the on-disk kubelet credential provider configuration to allow an optional tokenAttribute field to be configured. When it is set, the Kubelet will provision a token with the given audience bound to the current pod and its service account. This KSA token along with required annotations on the KSA defined in configuration will be sent to the credential provider plugin via its standard input (along with the image information that is already sent today). The KSA annotations to be sent are configurable in the kubelet credential provider configuration. (kubernetes/kubernetes#128372, @aramase) [SIG API Machinery, Auth, Node and Testing]

• Fixed the example validation rule in godoc:

  When configuring a JWT authenticator:

  If username.expression uses 'claims.email', then 'claims.email_verified' must be used in
  username.expression or extra[].valueExpression or claimValidationRules[].expression.
  An example claim validation rule expression that matches the validation automatically
  applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true) == true'.
  By explicitly comparing the value to true, we let type-checking see the result will be a boolean,
  and to make sure a non-boolean email_verified claim will be caught at runtime. (kubernetes/kubernetes#130875, @aramase) [SIG Auth and Release]

• For the InPlacePodVerticalScaling feature, the API server will no longer set the resize status to Proposed upon receiving a resize request. (kubernetes/kubernetes#130574, @natasha41575) [SIG Apps, Node and Testing]

• Graduate the MatchLabelKeys (MismatchLabelKeys) feature in PodAffinity (PodAntiAffinity) to GA (kubernetes/kubernetes#130463, @sanposhiho) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Graduated image volume sources to beta:

  • Allowed subPath/subPathExpr for image volumes
  • Added kubelet metrics kubelet_image_volume_requested_total, kubelet_image_volume_mounted_succeed_total and kubelet_image_volume_mounted_errors_total (kubernetes/kubernetes#130135, @saschagrunert) [SIG API Machinery, Apps, Node and Testing]

• Improved how the API server responds to list requests where the response format negotiates to Protobuf. List responses in Protobuf are marshalled one element at the time, drastically reducing memory needed to serve large collections. Streaming list responses can be disabled via the StreamingCollectionEncodingToProtobuf feature gate. (kubernetes/kubernetes#129407, @serathius) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Network, Node, Release, Scheduling, Storage and Testing]

• Introduced API type coordination.k8s.io/v1beta1/LeaseCandidate
  CoordinatedLeaderElection feature is Beta (kubernetes/kubernetes#130751, @Jefftree) [SIG API Machinery, Etcd and Testing]

• It introduces a new scope name VolumeAttributesClass.

  It matches all PVC objects that have the volume attributes class mentioned.

  If you want to limit the count of PVCs that have a specific volume attributes class. In that case, you can create a quota object with the scope name VolumeAttributesClass and a matchExpressions that match the volume attributes class. (kubernetes/kubernetes#124360, @carlory) [SIG API Machinery, Apps and Testing]

• Kubelet: add KubeletConfiguration.subidsPerPod (kubernetes/kubernetes#130028, @AkihiroSuda) [SIG API Machinery and Node]

• New configuration is introduced to the kubelet that allows it to track container images and the list of authentication information that lead to their successful pulls . This data is persisted across reboots of the host and restarts of the kubelet.

  The kubelet ensures any image requiring credential verification is always pulled if authentication information from an image pull is not yet present, thus enforcing authentication / re-authentication. This means an image pull might be attempted even in cases where a pod requests the IfNotPresent image pull policy, and might lead to the pod not starting if its pull policy is Never and is unable to present authentication information that lead to a previous successful pull of the image it is requesting. (kubernetes/kubernetes#128152, @stlaz) [SIG API Machinery, Architecture, Auth, Node and Testing]

• Promote JobSuccessPolicy E2E to Conformance (kubernetes/kubernetes#130658, @tenzen-y) [SIG API Machinery, Apps, Architecture and Testing]

• Promote NodeInclusionPolicyInPodTopologySpread to Stable in v1.33 (kubernetes/kubernetes#130920, @kerthcet) [SIG Apps, Node, Scheduling and Testing]

• Promote the JobSuccessPolicy to Stable. (kubernetes/kubernetes#130536, @tenzen-y) [SIG API Machinery, Apps, Architecture and Testing]

• Removed general available feature gate CPUManager. (kubernetes/kubernetes#129296, @carlory) [SIG API Machinery, Node and Testing]

• Start reporting swap capacity as part of node.status.nodeSystemInfo. (kubernetes/kubernetes#129954, @iholder101) [SIG API Machinery, Apps and Node]

• The ClusterTrustBundle API is moving to v1beta1.
  In order for the ClusterTrustBundleProjection feature to work on the kubelet side, the ClusterTrustBundle API must be available at v1beta1 version and the ClusterTrustBundleProjection feature gate must be enabled. If the API becomes later after kubelet started running, restart the kubelet to enable the feature. (kubernetes/kubernetes#128499, @stlaz) [SIG API Machinery, Apps, Auth, Etcd, Node, Storage and Testing]

• The Service trafficDistribution field, including the PreferClose option, has graduated
  to GA. Services that do not have the field configured will continue to operate
  with their existing behavior. Refer to the documentation
  https://kubernetes.io/docs/concepts/services-networking/service/#traffic-distribution
  for more details. (kubernetes/kubernetes#130673, @gauravkghildiyal) [SIG Apps, Network and Testing]

• The feature gate InPlacePodVerticalScalingAllocatedStatus is deprecated and no longer used. The AllocatedResources field in ContainerStatus is now guarded by the InPlacePodVerticalScaling feature gate. (kubernetes/kubernetes#130880, @tallclair) [SIG CLI, Node and Scheduling]

• The kube-controller-manager will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (kubernetes/kubernetes#130650, @natasha41575) [SIG API Machinery, Apps, Node, Scheduling, Storage, Testing and Windows]

• The kube-scheduler will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (kubernetes/kubernetes#130649, @natasha41575) [SIG Node, Scheduling and Testing]

• The kubelet will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (kubernetes/kubernetes#130573, @natasha41575) [SIG Apps, Node, Scheduling, Storage, Testing and Windows]

• The minimum value validation of ReplicationController's replicas and minReadySeconds fields have been migrated to declarative validation. The requiredness of both fields is also declaratively validated.
  If the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics.
  If the DeclarativeValidationTakeover feature gate is enabled, declarative validation is the primary source of errors for migrated fields. (kubernetes/kubernetes#130725, @jpbetz) [SIG API Machinery, Apps, Architecture, CLI, Cluster Lifecycle, Instrumentation, Network, Node and Storage]

• The resource.k8s.io/v1beta1 API is deprecated and will be removed in 1.36. Use v1beta2 instead. (kubernetes/kubernetes#129970, @mortent) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• Validation now requires new StatefulSets with a .spec.serviceName field value to pass DNS1123 validation. Previously created StatefulSets with an invalid .spec.serviceName field value could not create any pods, and should be deleted.

  • Published OpenAPI for the StatefulSet schema is corrected to indicate the .spec.serviceName is optional. (kubernetes/kubernetes#130233, @soltysh) [SIG API Machinery, Apps and Testing]

• When the ImprovedTrafficDistribution feature gate is enabled, a new
  trafficDistribution value PreferSameNode is available, which attempts to
  always route Service connections to an endpoint on the same node as
  the client. Additionally, PreferSameZone is introduced as an alias for
  PreferClose. (kubernetes/kubernetes#130844, @danwinship) [SIG API Machinery, Apps, Network and Windows]

• When the StrictIPCIDRValidation feature gate is enabled, Kubernetes will be
  slightly stricter about what values will be accepted as IP addresses and network
  address ranges (“CIDR blocks”).

  In particular, octets within IPv4 addresses are not allowed to have any leading
  0s, and IPv4-mapped IPv6 values (e.g. ::ffff:192.168.0.1) are forbidden.
  These sorts of values can potentially cause security problems when different
  components interpret the same string as referring to different IP addresses
  (as in CVE-2021-29923).

  This tightening applies only to fields in build-in API kinds, and not to
  custom resource kinds, values in Kubernetes configuration files, or
  command-line arguments.

  (When the feature gate is disabled, creating an object with such an invalid
  IP or CIDR value will result in a warning from the API server about the fact
  that it will be rejected in the future.) (kubernetes/kubernetes#122550, @danwinship) [SIG API Machinery, Apps, Network, Node, Scheduling and Testing]

• apidiscovery.k8s.io/v2beta1 API group is disabled by default (kubernetes/kubernetes#130347, @Jefftree) [SIG API Machinery and Testing]

• DRA support for a "one-of" prioritized list of selection criteria to satisfy a device request in a resource claim. (kubernetes/kubernetes#128586, @mortent) [SIG API Machinery, Apps, Etcd, Node, Scheduling and Testing]

• For the InPlacePodVerticalScaling feature, the API server will no longer set the resize status to Proposed upon receiving a resize request. (kubernetes/kubernetes#130574, @natasha41575) [SIG Apps, Node and Testing]

• The apiserver will now return warnings if you create objects with "invalid" IP or
  CIDR values (like "192.168.000.005", which should not have the extra zeros).
  Values with non-standard formats can introduce security problems, and will
  likely be forbidden in a future Kubernetes release. (kubernetes/kubernetes#128786, @danwinship) [SIG Apps, Network and Node]

• When the PodObservedGenerationTracking feature gate is set, the kubelet will populate status.observedGeneration to reflect the pod's latest metadata.generation that it has observed. (kubernetes/kubernetes#130352, @natasha41575) [SIG API Machinery, Apps, CLI, Node, Release, Scheduling, Storage, Testing and Windows]

• InPlacePodVerticalScaling: Memory limits cannot be decreased unless the memory resize restart policy is set to RestartContainer. Container resizePolicy is no longer mutable. (kubernetes/kubernetes#130183, @tallclair) [SIG Apps and Node]

• Introduced API type coordination.k8s.io/v1beta1/LeaseCandidate (kubernetes/kubernetes#130291, @Jefftree) [SIG API Machinery, Etcd and Testing]

• KEP-3857: Recursive Read-only (RRO) mounts: promote to GA (kubernetes/kubernetes#130116, @AkihiroSuda) [SIG Apps, Node and Testing]

• MergeDefaultEvictionSettings indicates that defaults for the evictionHard, evictionSoft, evictionSoftGracePeriod, and evictionMinimumReclaim fields should be merged into values specified for those fields in this configuration. Signals specified in this configuration take precedence. Signals not specified in this configuration inherit their defaults. (kubernetes/kubernetes#127577, @vaibhav2107) [SIG API Machinery and Node]

• Promote the Job's JobBackoffLimitPerIndex feature-gate to stable. (kubernetes/kubernetes#130061, @mimowo) [SIG API Machinery, Apps, Architecture and Testing]

• Promoted the feature gate AnyVolumeDataSource to GA. (kubernetes/kubernetes#129770, @sunnylovestiramisu) [SIG Apps, Storage and Testing]

• Added support for in-place vertical scaling of Pods with sidecars (containers defined within initContainers where the restartPolicy is Always). (kubernetes/kubernetes#128367, @vivzbansal) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]

• Kubectl: added alpha support for customizing kubectl behavior using preferences from a kuberc file (separate from kubeconfig). (kubernetes/kubernetes#125230, @ardaguclu) [SIG API Machinery, CLI and Testing]

• A new status field .status.terminatingReplicas is added to Deployments and ReplicaSets to allow tracking of terminating pods when the DeploymentReplicaSetTerminatingReplicas feature-gate is enabled. (kubernetes/kubernetes#128546, @atiratree) [SIG API Machinery, Apps and Testing]

• DRA API: the maximum number of pods which can use the same ResourceClaim is now 256 instead of 32. Beware that downgrading a cluster where this relaxed limit is in use to Kubernetes 1.32.0 is not supported because 1.32.0 would refuse to update ResourceClaims with more than 32 entries in the status.reservedFor field. (kubernetes/kubernetes#129543, @pohly) [SIG API Machinery, Node and Testing]

• DRA: CEL expressions using attribute strings exceeded the cost limit because their cost estimation was incomplete. (kubernetes/kubernetes#129661, @pohly) [SIG Node]

• DRA: when asking for "All" devices on a node, Kubernetes <= 1.32 proceeded to schedule pods onto nodes with no devices by not allocating any devices for those pods. Kubernetes 1.33 changes that to only picking nodes which have at least one device. Users who want the "proceed with scheduling also without devices" semantic can use the upcoming prioritized list feature with one sub-request for "all" devices and a second alternative with "count: 0". (kubernetes/kubernetes#129560, @bart0sh) [SIG API Machinery and Node]

• Graduate MultiCIDRServiceAllocator to stable and DisableAllocatorDualWrite to beta (disabled by default).
  Action required for Kubernetes distributions that manage the cluster Service CIDR.
  This feature allows users to define the cluster Service CIDR via a new API object: ServiceCIDR.
  Distributions or administrators of Kubernetes may want to control that new Service CIDRs added to the cluster does not overlap with other networks on the cluster, that only belong to a specific range of IPs or just simple retain the existing behavior of only having one ServiceCIDR per cluster. An example of a Validation Admission Policy to achieve this is:

  ---

  apiVersion: admissionregistration.k8s.io/v1
  kind: ValidatingAdmissionPolicy
  metadata:
  name: "servicecidrs.default"
  spec:
  failurePolicy: Fail
  matchConstraints:
  resourceRules:
  - apiGroups: ["networking.k8s.io"]
  apiVersions: ["v1","v1beta1"]
  operations: ["CREATE", "UPDATE"]
  resources: ["servicecidrs"]
  
  matchConditions:

  • name: 'exclude-default-servicecidr'
    expression: "object.metadata.name != 'kubernetes'"
    variables:
  • name: allowed
    expression: "['10.96.0.0/16','2001:db8::/64']"
    validations:
  • expression: "object.spec.cidrs.all(i , variables.allowed.exists(j , cidr(j).containsCIDR(i)))"

  ---

  apiVersion: admissionregistration.k8s.io/v1
  kind: ValidatingAdmissionPolicyBinding
  metadata:
  name: "servicecidrs-binding"
  spec:
  policyName: "servicecidrs.default"
  validationActions: [Deny,Audit]
  --- (kubernetes/kubernetes#128971, @aojea) [SIG Apps, Architecture, Auth, CLI, Etcd, Network, Release and Testing]

• Kubernetes starts validating NodeSelectorRequirement's values when creating pods. (kubernetes/kubernetes#128212, @AxeZhan) [SIG Apps and Scheduling]

• Kubernetes components that accept x509 client certificate authentication now read the user UID from a certificate subject name RDN with object id 1.3.6.1.4.1.57683.2. An RDN with this object id must contain a string value, and appear no more than once in the certificate subject. Reading the user UID from this RDN can be disabled by setting the beta feature gate AllowParsingUserUIDFromCertAuth to false (until the feature gate graduates to GA). (kubernetes/kubernetes#127897, @modulitos) [SIG API Machinery, Auth and Testing]

• Removed general available feature-gate PDBUnhealthyPodEvictionPolicy. (kubernetes/kubernetes#129500, @carlory) [SIG API Machinery, Apps and Auth]

• kubectl apply now coerces null values for labels and annotations in manifests to empty string values, consistent with typed JSON metadata decoding, rather than dropping all labels and annotations (kubernetes/kubernetes#129257, @liggitt) [SIG API Machinery]

v32.0.1

Kubernetes API Version: v1.32.2

Uncategorized Copy Section

• Adds support for providing cluster information to the exec credential provider if requested. (#2303, @brendandburns)
• Remove py from test dependencies (#2288, @jelly)

Bug or Regression Copy Section

• Fix dynamic client watch of named resource (#2076, @bobh66)
• Fixed PortForward proxy to close local Python sockets when the WebSocket closes. (#2316, @anvilpete)
• Fixes bug that would fail authentication when using the exec-provider with a specific cluster selected (#2340, @tomasaschan)

Feature Copy Section

• Add utility functions kubernetes.utils.duration.parse_duration and kubernetes.utils.duration.format_duration to manage Gateway API Duration strings as specified by GEP-2257. (#2261, @kflynn)
• Added the ability to use the optional apply parameter for functions within the utils.create_from_yaml submodule. This allows these functions to optionally use the DynamicClient.server_side_apply function to apply yaml manifests. (#2252, @dcmcand)
• Adding utils.format_quantity to convert decimal numbers into a canonical Kubernetes quantity. (#2216, @rkschamer)

v32.0.0

Kubernetes API Version: v1.32.1

Bug or Regression Copy Section

• Fixed PortForward proxy to close local Python sockets when the WebSocket closes. (#2316, @anvilpete)

v32.0.0b1

Kubernetes API Version: v1.32.1

API Change Copy Section

• DRA API: the maximum number of pods which can use the same ResourceClaim is now 256 instead of 32. Beware that downgrading a cluster where this relaxed limit is in use to Kubernetes 1.32.0 is not supported because 1.32.0 would refuse to update ResourceClaims with more than 32 entries in the status.reservedFor field. (kubernetes/kubernetes#129544, @pohly) [SIG API Machinery, Node and Testing]
• NONE (kubernetes/kubernetes#129598, @aravindhp) [SIG API Machinery and Node]

v32.0.0a1

Kubernetes API Version: v1.32.0

API Change Copy Section

• ACTION REQUIRED for custom scheduler plugin developers:
  PodEligibleToPreemptOthers in the preemption interface now includes ctx in the parameters.
  Please update your plugins' implementation accordingly. (kubernetes/kubernetes#126465, @googs1025) [SIG Scheduling]

• Changed NodeToStatusMap from a map to a struct and exposed methods to access the entries. Added absentNodesStatus, which informs the status of nodes that are absent in the map. For developers of out-of-tree PostFilter plugins, ensure to update the usage of NodeToStatusMap. Additionally, NodeToStatusMap should eventually be renamed to NodeToStatusReader. (kubernetes/kubernetes#126022, @macsko) [SIG Node, Scheduling, and Testing]

• A new /resize subresource was added to request pod resource resizing. Update your k8s client code to utilize the /resize subresource for Pod resizing operations. (kubernetes/kubernetes#128266, @AnishShah) [SIG API Machinery, Apps, Node and Testing]

• A new feature that allows unsafe deletion of corrupt resources has been added, it is disabled by default,
  and it can be enabled by setting the option --feature-gates=AllowUnsafeMalformedObjectDeletion=true.
  It comes with an API change, a new delete option ignoreStoreReadErrorWithClusterBreakingPotential has
  been introduced, it is not set by default, this maintains backward compatibility.
  In order to perform an unsafe deletion of a corrupt resource, the user must enable the option for the delete
  request. A resource is considered corrupt if it can not be successfully retrieved from the storage due to
  a) transformation error e.g. decryption failure, or b) the object failed to decode. Normal deletion flow is
  attempted first, and if it fails with a corrupt resource error then it triggers unsafe delete.
  In addition, when this feature is enabled, the 'details' field of 'Status' from the LIST response
  includes information that identifies the corrupt object(s).
  NOTE: unsafe deletion ignores finalizer constraints, and skips precondition checks.
  WARNING: this may break the workload associated with the resource being unsafe-deleted, if it relies on
  the normal deletion flow, so cluster breaking consequences apply. (kubernetes/kubernetes#127513, @tkashem) [SIG API Machinery, Etcd, Node and Testing]

• Added singleProcessOOMKill flag to the kubelet configuration. Setting that to true enable single process OOM killing in cgroups v2. In this mode, if a single process is OOM killed within a container, the remaining processes will not be OOM killed. (kubernetes/kubernetes#126096, @utam0k) [SIG API Machinery, Node, Testing and Windows]

• Added a /flagz endpoint for kube-apiserver endpoint. (kubernetes/kubernetes#127581, @richabanker) [SIG API Machinery, Architecture, Auth and Instrumentation]

• Added a Stream field to PodLogOptions, which allows clients to request certain log stream (stdout or stderr) of the container.
  Please also note that the combination of a specific Stream and TailLines is not supported. (kubernetes/kubernetes#127360, @knight42) [SIG API Machinery, Apps, Architecture, Node, Release and Testing]

• Added alpha support for asynchronous Pod preemption.
  When the SchedulerAsyncPreemption feature gate is enabled, the scheduler now runs API calls to trigger preemptions asynchronously for better performance. (kubernetes/kubernetes#128170, @sanposhiho) [SIG Scheduling and Testing]

• Added driver-owned fields in ResourceClaim.Status to report device status data for each allocated device. (kubernetes/kubernetes#128240, @LionelJouin) [SIG API Machinery, Network, Node and Testing]

• Added enforcement of an upper cost bound for DRA evaluations of CEL. The API server and scheduler now enforce an upper bound on the cost and runtime steps required for evaluating a CEL expression. (kubernetes/kubernetes#128101, @pohly) [SIG API Machinery and Node]

• Added the ability to change the maximum backoff delay accrued between container restarts for a node for containers in CrashLoopBackOff. To set this for a node, turn on the feature gate KubeletCrashLoopBackoffMax and set the CrashLoopBackOff.MaxContainerRestartPeriod field between "1s" and "300s" in your kubelet config file. (kubernetes/kubernetes#128374, @lauralorenz) [SIG API Machinery and Node]

• Allow for Pod search domains to be a single dot . or contain an underscore _ (kubernetes/kubernetes#127167, @adrianmoisey) [SIG Apps, Network and Testing]

• Annotation batch.kubernetes.io/cronjob-scheduled-timestamp added to Job objects scheduled from CronJobs is promoted to stable. (kubernetes/kubernetes#128336, @soltysh)

• Apply fsGroup policy for ReadWriteOncePod volumes. (kubernetes/kubernetes#128244, @gnufied) [SIG Storage and Testing]

• Changed the Pod API to support resources at spec level for pod-level resources. (kubernetes/kubernetes#128407, @ndixita) [SIG API Machinery, Apps, CLI, Cluster Lifecycle, Node, Release, Scheduling and Testing]

• ContainerStatus.AllocatedResources is now guarded by a separate feature gate, InPlacePodVerticalSaclingAllocatedStatus (kubernetes/kubernetes#128377, @tallclair) [SIG API Machinery, CLI, Node, Scheduling and Testing]

• Coordination.v1alpha1 API is dropped and replaced with coordination.v1alpha2. Old coordination.v1alpha1 types must be deleted before upgrade (kubernetes/kubernetes#127857, @Jefftree) [SIG API Machinery, Etcd, Scheduling and Testing]

• DRA: Restricted the length of opaque device configuration parameters. At admission time, Kubernetes enforces a 10KiB size limit. (kubernetes/kubernetes#128601, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• DRA: scheduling pods is up to 16x faster, depending on the scenario. Scheduling throughput depends a lot on cluster utilization. It is higher for lightly loaded clusters with free resources and gets lower when the cluster utilization increases. (kubernetes/kubernetes#127277, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]

• DRA: the DeviceRequestAllocationResult struct now has an "AdminAccess" field which should be used instead of the corresponding field in the DeviceRequest field when dealing with an allocation. If a device is only allocated for admin access, allocating it again for normal usage is now supported, as originally intended. To allow admin access, starting with 1.32 the DRAAdminAccess feature gate must be enabled. (kubernetes/kubernetes#127266, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Network, Node, Scheduling and Testing]

• Disallow k8s.io and kubernetes.io namespaced extra key in structured authentication configuration. (kubernetes/kubernetes#126553, @aramase) [SIG Auth]

• Fixed a bug in the NestedNumberAsFloat64 Unstructured field accessor that could have caused it to return rounded float64 values instead of errors when accessing very large int64 values. (kubernetes/kubernetes#128099, @benluddy)

• Fixed the bug where spec.terminationGracePeriodSeconds of the pod will always be overwritten by the MaxPodGracePeriodSeconds of the soft eviction, you can enable the AllowOverwriteTerminationGracePeriodSeconds feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you needed it. (kubernetes/kubernetes#122890, @HirazawaUi) [SIG API Machinery, Architecture, Node and Testing]

• Graduated Job's ManagedBy field to beta. (kubernetes/kubernetes#127402, @mimowo) [SIG API Machinery, Apps and Testing]

• Implemented a new, alpha seLinuxChangePolicy field within a Pod-level securityContext, under SELinuxChangePolicy feature gate. This field allows for opting out from mounting Pod volumes with SELinux label when SELinuxMount feature is enabled (it is alpha and disabled by default now).
  Please see the KEP how we expect to warn users before any SELinux behavior changes and how they can opt-out before. Note that this field and feature gate is useful only with clusters that run with SELinux enabled. No action is required on clusters without SELinux. (kubernetes/kubernetes#127981, @jsafrane) [SIG API Machinery, Apps, Architecture, Node, Storage and Testing]

• Introduced v1alpha1 API for mutating admission policies, enabling extensible # admission control via CEL expressions (KEP 3962: Mutating Admission Policies). # To use, enable the MutatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 # API via --runtime-config. (kubernetes/kubernetes#127134, @jpbetz) [SIG API Machinery, Auth, Etcd and Testing]

• Introduced compressible resource setting on system reserved and kube reserved slices. (kubernetes/kubernetes#125982, @harche)

• kube-apiserver: Promoted the StructuredAuthorizationConfiguration feature gate to GA. The --authorization-config flag now accepts AuthorizationConfiguration in version apiserver.config.k8s.io/v1 (with no changes from apiserver.config.k8s.io/v1beta1). (kubernetes/kubernetes#128172, @liggitt) [SIG API Machinery, Auth and Testing]

• kube-proxy now reconciles Service/Endpoint changes with conntrack table and cleans up only stale UDP flow entries (kubernetes/kubernetes#127318, @aroradaman) [SIG Network and Windows]

• kube-scheduler removed AzureDiskLimits ,CinderLimits EBSLimits and GCEPDLimits plugin. Given the corresponding CSI driver reports how many volumes a node can handle in NodeGetInfoResponse, the kubelet stores this limit in CSINode and the scheduler then knows the limit of the driver on the node. Removed plugins AzureDiskLimits, CinderLimits, EBSLimits and GCEPDLimits if you explicitly enabled them in the scheduler config. (kubernetes/kubernetes#124003, @carlory) [SIG Scheduling, Storage and Testing]

• kubelet: the --image-credential-provider-config file was loaded with strict deserialization, which failed if the config file contained duplicate or unknown fields. This protected against accidentally running with malformed config files, unindented files, or typos in field names, and it prevented unexpected behavior. (kubernetes/kubernetes#128062, @aramase) [SIG Auth and Node]

• NodeRestriction admission now validates the audience value that kubelet is requesting a service account token for is part of the pod spec volume. This change is introduced with a new kube-apiserver featuregate ServiceAccountNodeAudienceRestriction that's enabled by default. (kubernetes/kubernetes#128077, @aramase) [SIG Auth, Storage and Testing]

• Promoted CustomResourceFieldSelectors to stable; the feature was enabled by default. The --feature-gates=CustomResourceFieldSelectors=true flag was no longer needed on kube-apiserver binaries and would be removed in a future release. (kubernetes/kubernetes#127673, @jpbetz) [SIG API Machinery and Testing]

• Promoted feature gate StatefulSetAutoDeletePVC from beta to stable. (kubernetes/kubernetes#128247, @mattcary) [SIG API Machinery, Apps, Auth and Testing]

• Removed all support for classic dynamic resource allocation (DRA). The DRAControlPlaneController feature gate, formerly alpha, is no longer available. Kubernetes now only uses the structured parameters model (also alpha) for allocating dynamic resources to Pods.
  if and only if classic DRA was enabled in a cluster, remove all workloads (pods, app deployments, etc. ) which depend on classic DRA and make sure that all PodSchedulingContext resources are gone before upgrading. PodSchedulingContext resources cannot be removed through the apiserver after an upgrade and workloads would not work properly. (kubernetes/kubernetes#128003, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• Removed generally available feature gate HPAContainerMetrics (kubernetes/kubernetes#126862, @carlory) [SIG API Machinery, Apps and Autoscaling]

• Removed restrictions on subresource flag in kubectl commands (kubernetes/kubernetes#128296, @AnishShah) [SIG CLI]

• Revised the kubelet API Authorization with new subresources, that allow finer-grained authorization checks and access control for kubelet endpoints.
  Provided you enable the KubeletFineGrainedAuthz feature gate, you can access kubelet's /healthz endpoint by granting the caller nodes/helathz permission in RBAC.
  Similarly you can also access kubelet's /pods endpoint to fetch a list of Pods bound to that node by granting the caller nodes/pods permission in RBAC.
  Similarly you can also access kubelet's /configz endpoint to fetch kubelet's configuration by granting the caller nodes/configz permission in RBAC.
  You can still access kubelet's /healthz, /pods and /configz by granting the caller nodes/proxy permission in RBAC but that also grants the caller permissions to exec, run and attach to containers on the nodes and doing so does not follow the least privilege principle. Granting callers more permissions than they need can give attackers an opportunity to escalate privileges. (kubernetes/kubernetes#126347, @vinayakankugoyal) [SIG API Machinery, Auth, Cluster Lifecycle and Node]

• The core functionality of Dynamic Resource Allocation (DRA) got promoted to beta. No action is required when upgrading, the previous v1alpha3 API is still supported, so existing deployments and DRA drivers based on v1alpha3 continue to work. Downgrading from 1.32 to 1.31 with DRA resources in the cluster (resourceclaims, resourceclaimtemplates, deviceclasses, resourceslices) is not supported because the new v1beta1 is used as storage version and not readable by 1.31. (kubernetes/kubernetes#127511, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• The default value for node-monitor-grace-period has been increased to 50s (earlier 40s) (Ref - https://github.com/kubernetes/kubernetes/issues/121793) (kubernetes/kubernetes#126287, @devppratik) [SIG API Machinery, Apps and Node]

• The resource/v1alpha3.ResourceSliceList filed which should have been named "metadata" but was instead named "listMeta" is now properly "metadata". (kubernetes/kubernetes#126749, @thockin) [SIG API Machinery]

• The synthetic "Bookmark" event for the watch stream requests will now include a new annotation: kubernetes.io/initial-events-list-blueprint. THe annotation contains an empty, versioned list that is encoded in the requested format (such as protobuf, JSON, or CBOR), then base64-encoded and stored as a string. (kubernetes/kubernetes#127587, @p0lyn0mial) [SIG API Machinery]

• To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions.
  Name format CEL library is supported in new expressions. (kubernetes/kubernetes#126977, @aaron-prindle) [SIG API Machinery, Architecture, Auth, Etcd, Instrumentation, Release, Scheduling and Testing]

• Updated incorrect description of persistentVolumeClaimRetentionPolicy (kubernetes/kubernetes#126545, @yangjunmyfm192085) [SIG API Machinery, Apps and CLI]

• X.509 client certificate authentication to the kube-apiserver now produces credential IDs (derived from the certificate's signature) , for use in audit logging. (kubernetes/kubernetes#125634, @ahmedtd) [SIG API Machinery, Auth and Testing]

• Request header UID propagation is gated behind an alpha RemoteRequestHeaderUID feature gate. (kubernetes/kubernetes#129081, @stlaz) [SIG API Machinery, Cluster Lifecycle and Testing]

• A new /resize subresource was added to request pod resource resizing. Update your k8s client code to utilize the /resize subresource for Pod resizing operations. (kubernetes/kubernetes#128266, @AnishShah) [SIG API Machinery, Apps, Node and Testing]

• A new feature that allows unsafe deletion of corrupt resources has been added, it is disabled by default,
  and it can be enabled by setting the option --feature-gates=AllowUnsafeMalformedObjectDeletion=true.
  It comes with an API change, a new delete option ignoreStoreReadErrorWithClusterBreakingPotential has
  been introduced, it is not set by default, this maintains backward compatibility.
  In order to perform an unsafe deletion of a corrupt resource, the user must enable the option for the delete
  request. A resource is considered corrupt if it can not be successfully retrieved from the storage due to
  a) transformation error e.g. decryption failure, or b) the object failed to decode. Normal deletion flow is
  attempted first, and if it fails with a corrupt resource error then it triggers unsafe delete.
  In addition, when this feature is enabled, the 'details' field of 'Status' from the LIST response
  includes information that identifies the corrupt object(s).
  NOTE: unsafe deletion ignores finalizer constraints, and skips precondition checks.
  WARNING: this may break the workload associated with the resource being unsafe-deleted, if it relies on
  the normal deletion flow, so cluster breaking consequences apply. (kubernetes/kubernetes#127513, @tkashem) [SIG API Machinery, Etcd, Node and Testing]

• Add a Stream field to PodLogOptions, which allows clients to request certain log stream(stdout or stderr) of the container.
  Please also note that the combination of a specific Stream and TailLines is not supported. (kubernetes/kubernetes#127360, @knight42) [SIG API Machinery, Apps, Architecture, Node, Release and Testing]

• Add driver-owned fields in ResourceClaim.Status to report device status data for each allocated device. (kubernetes/kubernetes#128240, @LionelJouin) [SIG API Machinery, Network, Node and Testing]

• Added singleProcessOOMKill flag to the kubelet configuration. Setting that to true enable single process OOM killing in cgroups v2. In this mode, if a single process is OOM killed within a container, the remaining processes will not be OOM killed. (kubernetes/kubernetes#126096, @utam0k) [SIG API Machinery, Node, Testing and Windows]

• Added alpha support for asynchronous Pod preemption.
  When the SchedulerAsyncPreemption feature gate is enabled, the scheduler now runs API calls to trigger preemptions asynchronously for better performance. (kubernetes/kubernetes#128170, @sanposhiho) [SIG Scheduling and Testing]

• Added the ability to change the maximum backoff delay accrued between container restarts for a node for containers in CrashLoopBackOff. To set this for a node, turn on the feature gate KubeletCrashLoopBackoffMax and set the CrashLoopBackOff.MaxContainerRestartPeriod field between "1s" and "300s" in your kubelet config file. (kubernetes/kubernetes#128374, @lauralorenz) [SIG API Machinery and Node]

• Adds a /flagz endpoint for kube-apiserver endpoint (kubernetes/kubernetes#127581, @richabanker) [SIG API Machinery, Architecture, Auth and Instrumentation]

• Changed the Pod API to support resources at spec level for pod-level resources. (kubernetes/kubernetes#128407, @ndixita) [SIG API Machinery, Apps, CLI, Cluster Lifecycle, Node, Release, Scheduling and Testing]

• ContainerStatus.AllocatedResources is now guarded by a separate feature gate, InPlacePodVerticalSaclingAllocatedStatus (kubernetes/kubernetes#128377, @tallclair) [SIG API Machinery, CLI, Node, Scheduling and Testing]

• Coordination.v1alpha1 API is dropped and replaced with coordination.v1alpha2. Old coordination.v1alpha1 types must be deleted before upgrade (kubernetes/kubernetes#127857, @Jefftree) [SIG API Machinery, Etcd, Scheduling and Testing]

• DRA: Restricted the length of opaque device configuration parameters. At admission time, Kubernetes enforces a 10KiB size limit. (kubernetes/kubernetes#128601, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• Introduce v1alpha1 API for mutating admission policies, enabling extensible admission control via CEL expressions (KEP 3962: Mutating Admission Policies). To use, enable the MutatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config. (kubernetes/kubernetes#127134, @jpbetz) [SIG API Machinery, Auth, Etcd and Testing]

• NodeRestriction admission now validates the audience value that kubelet is requesting a service account token for is part of the pod spec volume. This change is introduced with a new kube-apiserver featuregate ServiceAccountNodeAudienceRestriction that's enabled by default. (kubernetes/kubernetes#128077, @aramase) [SIG Auth, Storage and Testing]

• Promoted feature gate StatefulSetAutoDeletePVC from beta to stable. (kubernetes/kubernetes#128247, @mattcary) [SIG API Machinery, Apps, Auth and Testing]

• Removed restrictions on subresource flag in kubectl commands (kubernetes/kubernetes#128296, @AnishShah) [SIG CLI]

• The core functionality of Dynamic Resource Allocation (DRA) got promoted to beta. No action is required when upgrading, the previous v1alpha3 API is still supported, so existing deployments and DRA drivers based on v1alpha3 continue to work. Downgrading from 1.32 to 1.31 with DRA resources in the cluster (resourceclaims, resourceclaimtemplates, deviceclasses, resourceslices) is not supported because the new v1beta1 is used as storage version and not readable by 1.31. (kubernetes/kubernetes#127511, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• DRA: scheduling pods is up to 16x faster, depending on the scenario. Scheduling throughput depends a lot on cluster utilization. It is higher for lightly loaded clusters with free resources and gets lower when the cluster utilization increases. (kubernetes/kubernetes#127277, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]

• DRA: the DeviceRequestAllocationResult struct now has an "AdminAccess" field which should be used instead of the corresponding field in the DeviceRequest field when dealing with an allocation. If a device is only allocated for admin access, allocating it again for normal usage is now supported, as originally intended. To allow admin access, starting with 1.32 the DRAAdminAccess feature gate must be enabled. (kubernetes/kubernetes#127266, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Network, Node, Scheduling and Testing]

• Implemented a new, alpha seLinuxChangePolicy field within a Pod-level securityContext, under SELinuxChangePolicy feature gate. This field allows for opting out from mounting Pod volumes with SELinux label when SELinuxMount feature is enabled (it is alpha and disabled by default now).
  Please see the KEP how we expect to warn users before any SELinux behavior changes and how they can opt-out before. Note that this field and feature gate is useful only with clusters that run with SELinux enabled. No action is required on clusters without SELinux. (kubernetes/kubernetes#127981, @jsafrane) [SIG API Machinery, Apps, Architecture, Node, Storage and Testing]

• Introduce v1alpha1 API for mutating admission policies, enabling extensible admission control via CEL expressions (KEP 3962: Mutating Admission Policies). To use, enable the MutatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config. (kubernetes/kubernetes#127134, @jpbetz) [SIG API Machinery, Auth, Etcd and Testing]

• Kube-proxy now reconciles Service/Endpoint changes with conntrack table and cleans up only stale UDP flow entries (kubernetes/kubernetes#127318, @aroradaman) [SIG Network and Windows]

• Removed generally available feature gate HPAContainerMetrics (kubernetes/kubernetes#126862, @carlory) [SIG API Machinery, Apps and Autoscaling]

• Added enforcement of an upper cost bound for DRA evaluations of CEL. The API server and scheduler now enforce an upper bound on the cost and runtime steps required for evaluating a CEL expression. (kubernetes/kubernetes#128101, @pohly) [SIG API Machinery and Node]

• Annotation batch.kubernetes.io/cronjob-scheduled-timestamp added to Job objects scheduled from CronJobs is promoted to stable (kubernetes/kubernetes#128336, @soltysh) [SIG Apps]

• Apply fsGroup policy for ReadWriteOncePod volumes (kubernetes/kubernetes#128244, @gnufied) [SIG Storage and Testing]

• Graduate Job's ManagedBy field to Beta (kubernetes/kubernetes#127402, @mimowo) [SIG API Machinery, Apps and Testing]

• Kube-apiserver: Promoted the StructuredAuthorizationConfiguration feature gate to GA. The --authorization-config flag now accepts AuthorizationConfiguration in version apiserver.config.k8s.io/v1 (with no changes from apiserver.config.k8s.io/v1beta1). (kubernetes/kubernetes#128172, @liggitt) [SIG API Machinery, Auth and Testing]

• Removed all support for classic dynamic resource allocation (DRA). The DRAControlPlaneController feature gate, formerly alpha, is no longer available. Kubernetes now only uses the structured parameters model (also alpha) for allocating dynamic resources to Pods.

  if and only if classic DRA was enabled in a cluster, remove all workloads (pods, app deployments, etc. ) which depend on classic DRA and make sure that all PodSchedulingContext resources are gone before upgrading. PodSchedulingContext resources cannot be removed through the apiserver after an upgrade and workloads would not work properly. (kubernetes/kubernetes#128003, @pohly) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• Revised the Kubelet API Authorization with new subresources, that allow finer-grained authorization checks and access control for kubelet endpoints.
  Provided you enable the KubeletFineGrainedAuthz feature gate, you can access kubelet's /healthz endpoint by granting the caller nodes/helathz permission in RBAC.
  Similarly you can also access kubelet's /pods endpoint to fetch a list of Pods bound to that node by granting the caller nodes/pods permission in RBAC.
  Similarly you can also access kubelet's /configz endpoint to fetch kubelet's configuration by granting the caller nodes/configz permission in RBAC.
  You can still access kubelet's /healthz, /pods and /configz by granting the caller nodes/proxy permission in RBAC but that also grants the caller permissions to exec, run and attach to containers on the nodes and doing so does not follow the least privilege principle. Granting callers more permissions than they need can give attackers an opportunity to escalate privileges. (kubernetes/kubernetes#126347, @vinayakankugoyal) [SIG API Machinery, Auth, Cluster Lifecycle and Node]

• Fixed a bug in the NestedNumberAsFloat64 Unstructured field accessor that could cause it to return rounded float64 values instead of errors when accessing very large int64 values. (kubernetes/kubernetes#128099, @benluddy) [SIG API Machinery]

• Introduce compressible resource setting on system reserved and kube reserved slices (kubernetes/kubernetes#125982, @harche) [SIG Node]

• Kubelet: the --image-credential-provider-config file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (kubernetes/kubernetes#128062, @aramase) [SIG Auth and Node]

• Promoted CustomResourceFieldSelectors to stable; the feature is enabled by default. --feature-gates=CustomResourceFieldSelectors=true not needed on kube-apiserver binaries and will be removed in a future release. (kubernetes/kubernetes#127673, @jpbetz) [SIG API Machinery and Testing]

• ACTION REQUIRED for custom scheduler plugin developers:

  • PodEligibleToPreemptOthers in the preemption interface gets ctx in the parameters.
    Please change your plugins' implementation accordingly. (kubernetes/kubernetes#126465, @googs1025) [SIG Scheduling]
  • Changed NodeToStatusMap from map to struct and exposed methods to access the entries. Added absentNodesStatus, which inform what is the status of nodes that are absent in the map.
  • For developers of out-of-tree PostFilter plugins, make sure to update usage of NodeToStatusMap. Additionally, NodeToStatusMap should be eventually renamed to NodeToStatusReader. (kubernetes/kubernetes#126022, @macsko) [SIG Node, Scheduling and Testing]

• Allow for Pod search domains to be a single dot "." or contain an underscore "_" (kubernetes/kubernetes#127167, @adrianmoisey) [SIG Apps, Network and Testing]

• Disallow k8s.io and kubernetes.io namespaced extra key in structured authentication configuration. (kubernetes/kubernetes#126553, @aramase) [SIG Auth]

• Fix the bug where spec.terminationGracePeriodSeconds of the pod will always be overwritten by the MaxPodGracePeriodSeconds of the soft eviction, you can enable the AllowOverwriteTerminationGracePeriodSeconds feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you need it. (kubernetes/kubernetes#122890, @HirazawaUi) [SIG API Machinery, Architecture, Node and Testing]

• Kube-scheduler removed the following plugins:

  • AzureDiskLimits
  • CinderLimits
  • EBSLimits
  • GCEPDLimits
    Because the corresponding CSI driver reports how many volumes a node can handle in NodeGetInfoResponse, the kubelet stores this limit in CSINode and the scheduler then knows the driver's limit on the node.
    Remove plugins AzureDiskLimits, CinderLimits, EBSLimits and GCEPDLimits if you explicitly enabled them in the scheduler config. (kubernetes/kubernetes#124003, @carlory) [SIG Scheduling, Storage and Testing]

• Promoted CustomResourceFieldSelectors to stable; the feature is enabled by default. --feature-gates=CustomResourceFieldSelectors=true not needed on kube-apiserver binaries and will be removed in a future release. (kubernetes/kubernetes#127673, @jpbetz) [SIG API Machinery and Testing]

• The default value for node-monitor-grace-period has been increased to 50s (earlier 40s) (Ref - https://github.com/kubernetes/kubernetes/issues/121793) (kubernetes/kubernetes#126287, @devppratik) [SIG API Machinery, Apps and Node]

• The resource/v1alpha3.ResourceSliceList filed which should have been named "metadata" but was instead named "listMeta" is now properly "metadata". (kubernetes/kubernetes#126749, @thockin) [SIG API Machinery]

• The synthetic "Bookmark" event for the watch stream requests will now include a new annotation: kubernetes.io/initial-events-list-blueprint. THe annotation contains an empty, versioned list that is encoded in the requested format (such as protobuf, JSON, or CBOR), then base64-encoded and stored as a string. (kubernetes/kubernetes#127587, @p0lyn0mial) [SIG API Machinery]

• To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions.
  Name format CEL library is supported in new expressions. (kubernetes/kubernetes#126977, @aaron-prindle) [SIG API Machinery, Architecture, Auth, Etcd, Instrumentation, Release, Scheduling and Testing]

• Updated incorrect description of persistentVolumeClaimRetentionPolicy (kubernetes/kubernetes#126545, @yangjunmyfm192085) [SIG API Machinery, Apps and CLI]

• X.509 client certificate authentication to kube-apiserver now produces credential IDs (derived from the certificate's signature) for use by audit logging. (kubernetes/kubernetes#125634, @ahmedtd) [SIG API Machinery, Auth and Testing]

v31.0.0

Kubernetes API Version: v1.31.0

v31.0.0b1

Kubernetes API Version: v1.31.0

v31.0.0a1

Kubernetes API Version: v1.31.0

API Change Copy Section

• 'ACTION REQUIRED: The Dynamic Resource Allocation (DRA) driver's DaemonSet
  must be deployed with a service account that enables writing ResourceSlice
  and reading ResourceClaim objects.'
  (kubernetes/kubernetes#125163, @pohly) [SIG Auth, Node and Testing]

• Add UserNamespaces field to NodeRuntimeHandlerFeatures (kubernetes/kubernetes#126034, @sohankunkerkar) [SIG API Machinery, Apps and Node]

• Added Coordinated Leader Election as Alpha under the CoordinatedLeaderElection feature gate. With the feature enabled, the control plane can use LeaseCandidate objects (coordination.k8s.io/v1alpha1 API group) to participate in a leader election and let the kube-apiserver select the best instance according to some strategy. (kubernetes/kubernetes#124012, @Jefftree) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Release, Scheduling and Testing]

• Added a .status.features.supplementalGroupsPolicy field to Nodes. The field is true when the feature is implemented in the CRI implementation (KEP-3619). (kubernetes/kubernetes#125470, @everpeace) [SIG API Machinery, Apps, Node and Testing]

• Added an allocatedResourcesStatus to each container status to indicate the health status of devices exposed by the device plugin. (kubernetes/kubernetes#126243, @SergeyKanzhelev) [SIG API Machinery, Apps, Node and Testing]

• Added support to the kube-proxy nodePortAddresses / --nodeport-addresses option to
  accept the value "primary", meaning to only listen for NodePort connections
  on the node's primary IPv4 and/or IPv6 address (according to the Node object).
  This is strongly recommended, if you were not previously using
  --nodeport-addresses, to avoid surprising behavior.
  (This behavior is enabled by default with the nftables backend; you would
  need to explicitly request --nodeport-addresses 0.0.0.0/0,::/0 there to get
  the traditional "listen on all interfaces" behavior.) (kubernetes/kubernetes#123105, @danwinship) [SIG API Machinery, Network and Windows]

• Added the feature gates StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks to enforce the strict cost calculation for CEL extended libraries. It is strongly recommended to turn on the feature gates as early as possible. (kubernetes/kubernetes#124675, @cici37) [SIG API Machinery, Auth, Node and Testing]

• Changed how the API server handles updates to .spec.defaultBackend of Ingress objects.
  Server-side apply now considers .spec.defaultBackend to be an atomic struct. This means that any field-owner who sets values in that struct (they are mutually exclusive) owns the whole struct. For almost all users this change has no impact; for controllers that want to change the default backend port from number to name (or vice-versa), this makes it easier. (kubernetes/kubernetes#126207, @thockin) [SIG API Machinery]

• Component-base/logs: when compiled with Go >= 1.21, component-base will automatically configure the slog default logger together with initializing klog. (kubernetes/kubernetes#120696, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing]

• CustomResourceDefinition objects created with non-empty caBundle fields which are invalid or do not contain any certificates will not appear in discovery or serve endpoints until a valid caBundle is provided. Updates to CustomResourceDefinition are no longer allowed to transition a valid caBundle field to an invalid caBundle field, because this breaks serving of the existing CustomResourceDefinition. (kubernetes/kubernetes#124061, @Jefftree) [SIG API Machinery]

• Dynamic Resource Allocation (DRA): Added a feature so the number of ResourceClaim objects can be limited per namespace and by the number of devices requested through a specific class via the v1.ResourceQuota mechanism. (kubernetes/kubernetes#120611, @pohly) [SIG API Machinery, Apps, Auth, CLI, Etcd, Node, Release, Scheduling and Testing]

• Dynamic Resource Allocation (DRA): client-side validation of a ResourceHandle would have accepted a missing DriverName, whereas server-side validation then would have raised an error. (kubernetes/kubernetes#124075, @pohly)

• Dynamic Resource Allocation (DRA): in the pod.spec.recourceClaims array, the source indirection is no longer necessary. Instead of e.g. source: resourceClaimTemplateName: my-template, one can write resourceClaimTemplateName: my-template. (kubernetes/kubernetes#125116, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• Enhanced the Dynamic Resource Allocation (DRA) with an updated version of the resource.k8s.io API group. The primary user-facing type remains the ResourceClaim, however significant changes have been made, resulting in the new version, v1alpha3, which is not compatible with the previous version. (kubernetes/kubernetes#125488, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Etcd, Node, Release, Scheduling, Storage and Testing]

• Fixed a 1.30.0 regression in OpenAPI descriptions of the imagePullSecrets and
  hostAliases fields to mark the fields used as keys in those lists as either defaulted
  or required. (kubernetes/kubernetes#124553, @pmalek)

• Fixed a 1.30.0 regression in openapi descriptions of PodIP.IP and HostIP.IP fields to mark the fields used as keys in those lists as required. (kubernetes/kubernetes#126057, @thockin)

• Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an items field. (kubernetes/kubernetes#124568, @xyz-li) [SIG API Machinery]

• Fixed a deep copy issue when retrieving the controller reference. (kubernetes/kubernetes#124116, @HiranmoyChowdhury) [SIG API Machinery and Release]

• Fixed code-generator client-gen to work with api/v1-like package structure. (kubernetes/kubernetes#125162, @sttts) [SIG API Machinery and Apps]

• Fixed incorrect "v1 Binding is deprecated in v1.6+" warning in kube-scheduler log. (kubernetes/kubernetes#125540, @pohly) [SIG API Machinery]

• Fixed the comment for the Job's managedBy field. (kubernetes/kubernetes#124793, @mimowo) [SIG API Machinery and Apps]

• Fixed the documentation for the default value of the procMount entry in securityContext within a Pod.
  The documentation was previously using the name of the internal variable DefaultProcMount, rather than the actual value, "Default". (kubernetes/kubernetes#125782, @aborrero) [SIG Apps and Node]

• Graduate PodDisruptionConditions to GA and lock (kubernetes/kubernetes#125461, @mimowo) [SIG Apps, Node, Scheduling and Testing]

• Graduated MatchLabelKeys/MismatchLabelKeys feature in PodAffinity/PodAntiAffinity to Beta. (kubernetes/kubernetes#123638, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing]

• Graduated JobPodFailurePolicy to GA and locked it to it's default. (kubernetes/kubernetes#125442, @mimowo) [SIG API Machinery, Apps, Scheduling and Testing]

• Graduated the Job successPolicy field to beta.

  The new reason label, "SuccessPolicy" and "CompletionsReached" are added to the "jobs_finished_total" metric.
  Additionally, if you enable the JobSuccessPolicy feature gate, the Job gets "CompletionsReached" reason for the "SuccessCriteriaMet" and "Complete" condition type
  when the number of succeeded Job Pods (.status.succeeded) reached the desired completions (.spec.completions). (kubernetes/kubernetes#126067, @tenzen-y) [SIG API Machinery, Apps and Testing]

• Graduated the DisableNodeKubeProxyVersion feature gate to beta. By default, the kubelet no longer attempts to set the .status.kubeProxyVersion field for its associated Node. (kubernetes/kubernetes#123845, @HirazawaUi) [SIG API Machinery, Cloud Provider, Network, Node and Testing]

• Improved scheduling performance when many nodes, and prefilter returned 1-2 nodes (e.g. daemonset)

  For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status. (kubernetes/kubernetes#125197, @gabesaba)

• Introduced a new boolean kubelet flag --fail-cgroupv1. (kubernetes/kubernetes#126031, @harche) [SIG API Machinery and Node]

• K8s.io/apimachinery/pkg/util/runtime: Added support for new calls to handle panics and errors in the context where they occur. PanicHandlers and ErrorHandlers now must accept a context parameter for that. Log output is structured instead of unstructured. (kubernetes/kubernetes#121970, @pohly) [SIG API Machinery and Instrumentation]

• KEP-1880: Users of the new feature to add multiple service CIDR will use by default a dual-write strategy on the new ClusterIP allocators to avoid the problem of possible duplicate IPs allocated to Services when running skewed kube-apiservers using different allocators. They can opt-out of this behavior by enabled the feature gate DisableAllocatorDualWrite. (kubernetes/kubernetes#122047, @aojea) [SIG API Machinery, Apps, Instrumentation and Testing]

• Kube-apiserver: Added Alpha features to allow API server authz to check the context of requests:

  • The AuthorizeWithSelectors feature gate enables including field and label selector information from requests in webhook authorization calls.
  • The AuthorizeNodeWithSelectors feature gate changes node authorizer behavior to limit requests from node API clients, so that each Node can only get / list / watch its own Node API object, and can also only get / list / watch Pod API objects bound to that node. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or obtain broader read access independent of the node authorizer. (kubernetes/kubernetes#125571, @liggitt) [SIG API Machinery, Auth, Node, Scheduling and Testing]

• Kube-apiserver: ControllerRevision objects are now verified to contain valid JSON data in the data field. (kubernetes/kubernetes#125549, @liggitt) [SIG API Machinery and Apps]

• Kube-apiserver: the --encryption-provider-config file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. When --encryption-provider-config-automatic-reload is used, new encryption config files that contain typos after the kube-apiserver is running are treated as invalid and the last valid config is used. (kubernetes/kubernetes#124912, @enj) [SIG API Machinery and Auth]

• Kube-controller-manager: the horizontal-pod-autoscaler-upscale-delay and horizontal-pod-autoscaler-downscale-delay flags have been removed (deprecated and non-functional since v1.12). (kubernetes/kubernetes#124948, @SataQiu) [SIG API Machinery, Apps and Autoscaling]

• Made kube-proxy Windows service control manager integration (--windows-service) configurable in v1alpha1 component configuration via windowsRunAsService field. (kubernetes/kubernetes#126072, @aroradaman) [SIG Network and Scalability]

• PersistentVolumeLastPhaseTransitionTime feature is stable and enabled by default. (kubernetes/kubernetes#124969, @RomanBednar) [SIG API Machinery, Apps, Storage and Testing]

• Promoted LocalStorageCapacityIsolation to beta; the behaviour is enabled by default. Within the kubelet, storage capacity isolation is active if the feature gate is enabled and the specific Pod is using a user namespace. (kubernetes/kubernetes#126014, @PannagaRao) [SIG Apps, Autoscaling, Node, Storage and Testing]

• Promoted StatefulSetStartOrdinal to stable. This means --feature-gates=StatefulSetStartOrdinal=true are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation. (kubernetes/kubernetes#125374, @pwschuurman) [SIG API Machinery, Apps and Testing]

• Promoted feature-gate VolumeAttributesClass to beta (disabled by default). Users need to enable the feature gate and the storage.k8s.io/v1beta1 API group to use this feature.
  Promoted the VolumeAttributesClass API to beta. (kubernetes/kubernetes#126145, @carlory) [SIG API Machinery, Apps, CLI, Etcd, Storage and Testing]

• Removed deprecated command flags --volume-host-cidr-denylist
  and --volume-host-allow-local-loopback from kube-controller-manager.
  (kubernetes/kubernetes#124017, @carlory) [SIG API Machinery, Apps, Cloud Provider and Storage]

• Removed feature gate CustomResourceValidationExpressions. (kubernetes/kubernetes#126136, @cici37) [SIG API Machinery, Cloud Provider and Testing]

• Reverted a change where ConsistentListFromCache was moved to beta and enabled by default. (kubernetes/kubernetes#126139, @enj)

• Revised the Pod API with Alpha support for volumes derived from OCI artifacts. This feature is behind the ImageVolume feature gate. (kubernetes/kubernetes#125660, @saschagrunert) [SIG API Machinery, Apps and Node]

• Supported fine-grained supplemental groups policy (KEP-3619), which enabled
  fine-grained control for supplementary groups in the first container processes.
  This allows you to choose whether to include groups defined in the container image (/etc/groups)
  for the container's primary UID or not. (kubernetes/kubernetes#117842, @everpeace) [SIG API Machinery, Apps and Node]

• The (alpha) nftables mode of kube-proxy now requires version 1.0.1 or later
  of the nft command-line, and kernel 5.13 or later. (For testing/development
  purposes, you can use older kernels, as far back as 5.4, if you set the
  nftables.skipKernelVersionCheck option in the kube-proxy config, but this is not
  recommended in production since it may cause problems with other nftables
  users on the system.) (kubernetes/kubernetes#124152, @danwinship) [SIG Network]

• To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions for existing expressions in storage, will fully support runtime in next release for compatibility concern. (kubernetes/kubernetes#126188, @cici37) [SIG API Machinery and Testing]

• Updated the feature MultiCIDRServiceAllocator to beta (disabled by default). Users need to enable the feature gate and the networking v1beta1 group to be able to use this new feature, that allows to dynamically reconfigure Service CIDR ranges. (kubernetes/kubernetes#125021, @aojea) [SIG API Machinery, Apps, CLI, Etcd, Instrumentation, Network and Testing]

• Use omitempty for optional Job Pod Failure Policy fields. (kubernetes/kubernetes#126046, @mimowo)

• User can choose a different static policy option SpreadPhysicalCPUsPreferredOption to spread cpus across physical cpus for some specific applications (kubernetes/kubernetes#123733, @Jeffwan) [SIG Node]

• When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed. (kubernetes/kubernetes#124917, @vinayakankugoyal) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]

• Move ConsistentListFromCache feature flag to Beta and enable it by default (kubernetes/kubernetes#126469, @serathius) [SIG API Machinery]

• Add Coordinated Leader Election as alpha under the CoordinatedLeaderElection feature gate. With the feature enabled, the control plane can use LeaseCandidate objects (coordination.k8s.io/v1alpha1 API group) to participate in a leader election and let the kube-apiserver select the best instance according to some strategy. (kubernetes/kubernetes#124012, @Jefftree) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Release, Scheduling and Testing]

• Add an AllocatedResourcesStatus to each container status to indicate the health status of devices exposed by the device plugin. (kubernetes/kubernetes#126243, @SergeyKanzhelev) [SIG API Machinery, Apps, Node and Testing]

• Added Node.Status.Features.SupplementalGroupsPolicy field which is set to true when the feature is implemented in the CRI implementation (KEP-3619) (kubernetes/kubernetes#125470, @everpeace) [SIG API Machinery, Apps, Node and Testing]

• CustomResourceDefinition objects created with non-empty caBundle fields which are invalid or do not contain any certificates will not appear in discovery or serve endpoints until a valid caBundle is provided. Updates to CustomResourceDefinition are no longer allowed to transition a valid caBundle field to an invalid caBundle field. (kubernetes/kubernetes#124061, @Jefftree) [SIG API Machinery]

• DRA: The DRA driver's daemonset must be deployed with a service account that enables writing ResourceSlice and reading ResourceClaim objects. (kubernetes/kubernetes#125163, @pohly) [SIG Auth, Node and Testing]

• DRA: new API and several new features (kubernetes/kubernetes#125488, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Etcd, Node, Release, Scheduling, Storage and Testing]

• DRA: the number of ResourceClaim objects can be limited per namespace and by the number of devices requested through a specific class via the v1.ResourceQuota mechanism. (kubernetes/kubernetes#120611, @pohly) [SIG API Machinery, Apps, Auth, CLI, Etcd, Node, Release, Scheduling and Testing]

• Fix the documentation for the default value of the procMount entry in the pod securityContext.
  The documentation was previously using the name of the internal variable 'DefaultProcMount' rather than the actual value 'Default'. (kubernetes/kubernetes#125782, @aborrero) [SIG Apps and Node]

• Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an items field. (kubernetes/kubernetes#124568, @xyz-li) [SIG API Machinery]

• Graduate the Job SuccessPolicy to Beta.

  The new reason label, "SuccessPolicy" and "CompletionsReached" are added to the "jobs_finished_total" metric.
  Additionally, If we enable the "JobSuccessPolicy" feature gate, the Job gets "CompletionsReached" reason for the "SuccessCriteriaMet" and "Complete" condition type
  when the number of succeeded Job Pods (".status.succeeded") reached the desired completions (".spec.completions"). (kubernetes/kubernetes#126067, @tenzen-y) [SIG API Machinery, Apps and Testing]

• Introduce a new boolean kubelet flag --fail-cgroupv1 (kubernetes/kubernetes#126031, @harche) [SIG API Machinery and Node]

• Kube-apiserver: adds an alpha AuthorizeWithSelectors feature that includes field and label selector information from requests in webhook authorization calls; adds an alpha AuthorizeNodeWithSelectors feature that makes the node authorizer limit requests from node API clients to get / list / watch its own Node API object, and to get / list / watch its own Pod API objects. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or grant broader read access independent of the node authorizer. (kubernetes/kubernetes#125571, @liggitt) [SIG API Machinery, Auth, Node, Scheduling and Testing]

• Kube-proxy Windows service control manager integration(--windows-service) is now configurable in v1alpha1 component configuration via WindowsRunAsService field (kubernetes/kubernetes#126072, @aroradaman) [SIG Network and Scalability]

• Promote LocalStorageCapacityIsolation to beta and enable if user namespace is enabled for the pod (kubernetes/kubernetes#126014, @PannagaRao) [SIG Apps, Autoscaling, Node, Storage and Testing]

• Promote StatefulSetStartOrdinal to stable. This means --feature-gates=StatefulSetStartOrdinal=true are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation (kubernetes/kubernetes#125374, @pwschuurman) [SIG API Machinery, Apps and Testing]

• Promoted feature-gate VolumeAttributesClass to beta (disabled by default). Users need to enable the feature gate and the storage v1beta1 group to use this new feature.

  • Promoted API VolumeAttributesClass and VolumeAttributesClassList to storage.k8s.io/v1beta1. (kubernetes/kubernetes#126145, @carlory) [SIG API Machinery, Apps, CLI, Etcd, Storage and Testing]

• Removed feature gate CustomResourceValidationExpressions. (kubernetes/kubernetes#126136, @cici37) [SIG API Machinery, Cloud Provider and Testing]

• Revert "Move ConsistentListFromCache feature flag to Beta and enable it by default" (kubernetes/kubernetes#126139, @enj) [SIG API Machinery]

• Revised the Pod API with alpha support for volumes derived from OCI artefacts.
  This feature is behind the ImageVolume feature gate. (kubernetes/kubernetes#125660, @saschagrunert) [SIG API Machinery, Apps and Node]

• The Ingress.spec.defaultBackend is now considered an atomic struct for the purposes of server-side-apply. This means that any field-owner who sets values in that struct (they are mutually exclusive) owns the whole struct. For almost all users this change has no impact. For controllers which want to change port from number to name (or vice-versa), this makes it easier. (kubernetes/kubernetes#126207, @thockin) [SIG API Machinery]

• To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions for existing expressions in storage, will fully support runtime in next release for compatibility concern. (kubernetes/kubernetes#126188, @cici37) [SIG API Machinery and Testing]

• Add UserNamespaces field to NodeRuntimeHandlerFeatures (kubernetes/kubernetes#126034, @sohankunkerkar) [SIG API Machinery, Apps and Node]

• Fixes a 1.30.0 regression in openapi descriptions of PodIP.IP and HostIP.IP fields to mark the fields used as keys in those lists as required. (kubernetes/kubernetes#126057, @thockin) [SIG API Machinery]

• Graduate JobPodFailurePolicy to GA and lock (kubernetes/kubernetes#125442, @mimowo) [SIG API Machinery, Apps, Scheduling and Testing]

• Graduate PodDisruptionConditions to GA and lock (kubernetes/kubernetes#125461, @mimowo) [SIG Apps, Node, Scheduling and Testing]

• PersistentVolumeLastPhaseTransitionTime feature is stable and enabled by default. (kubernetes/kubernetes#124969, @RomanBednar) [SIG API Machinery, Apps, Storage and Testing]

• The (alpha) nftables mode of kube-proxy now requires version 1.0.1 or later
  of the nft command-line, and kernel 5.13 or later. (For testing/development
  purposes, you can use older kernels, as far back as 5.4, if you set the
  nftables.skipKernelVersionCheck option in the kube-proxy config, but this is not
  recommended in production since it may cause problems with other nftables
  users on the system.) (kubernetes/kubernetes#124152, @danwinship) [SIG Network]

• Use omitempty for optional Job Pod Failure Policy fields (kubernetes/kubernetes#126046, @mimowo) [SIG Apps]

• User can choose a different static policy option SpreadPhysicalCPUsPreferredOption to spread cpus across physical cpus for some specific applications (kubernetes/kubernetes#123733, @Jeffwan) [SIG Node]

• DRA: in the pod.spec.recourceClaims array, the source indirection is no longer necessary. Instead of e.g. source: resourceClaimTemplateName: my-template, one can write resourceClaimTemplateName: my-template. (kubernetes/kubernetes#125116, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• Fix code-generator client-gen to work with api/v1-like package structure. (kubernetes/kubernetes#125162, @sttts) [SIG API Machinery and Apps]

• KEP-1880: Users of the new feature to add multiple service CIDR will use by default a dual-write strategy on the new ClusterIP allocators to avoid the problem of possible duplicate IPs allocated to Services when running skewed kube-apiservers using different allocators. They can opt-out of this behavior by enabled the feature gate DisableAllocatorDualWrite (kubernetes/kubernetes#122047, @aojea) [SIG API Machinery, Apps, Instrumentation and Testing]

• Kube-apiserver: ControllerRevision objects are now verified to contain valid JSON data in the data field. (kubernetes/kubernetes#125549, @liggitt) [SIG API Machinery and Apps]

• Update the feature MultiCIDRServiceAllocator to beta (disabled by default). Users need to enable the feature gate and the networking v1beta1 group to be able to use this new feature, that allows to dynamically reconfigure Service CIDR ranges. (kubernetes/kubernetes#125021, @aojea) [SIG API Machinery, Apps, CLI, Etcd, Instrumentation, Network and Testing]

• When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed. (kubernetes/kubernetes#124917, @vinayakankugoyal) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]

• Fixed incorrect "v1 Binding is deprecated in v1.6+" warning in kube-scheduler log. (kubernetes/kubernetes#125540, @pohly) [SIG API Machinery]

• Added the feature gates StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks to enforce the strct cost calculation for CEL extended libraries. It is strongly recommended to turn on the feature gates as early as possible. (kubernetes/kubernetes#124675, @cici37) [SIG API Machinery, Auth, Node and Testing]

• Component-base/logs: when compiled with Go >= 1.21, component-base will automatically configure the slog default logger together with initializing klog. (kubernetes/kubernetes#120696, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing]

• DRA: client-side validation of a ResourceHandle would have accepted a missing DriverName, whereas server-side validation then would have raised an error. (kubernetes/kubernetes#124075, @pohly) [SIG Apps]

• Fix Deep Copy issue in getting controller reference (kubernetes/kubernetes#124116, @HiranmoyChowdhury) [SIG API Machinery and Release]

• Fix the comment for the Job's managedBy field (kubernetes/kubernetes#124793, @mimowo) [SIG API Machinery and Apps]

• Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. (kubernetes/kubernetes#124553, @pmalek) [SIG API Machinery]

• Graduate MatchLabelKeys/MismatchLabelKeys feature in PodAffinity/PodAntiAffinity to Beta (kubernetes/kubernetes#123638, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing]

• Graduated the DisableNodeKubeProxyVersion feature gate to beta. By default, the kubelet no longer attempts to set the .status.kubeProxyVersion field for its associated Node. (kubernetes/kubernetes#123845, @HirazawaUi) [SIG API Machinery, Cloud Provider, Network, Node and Testing]

• Improved scheduling performance when many nodes, and prefilter returns 1-2 nodes (e.g. daemonset)

  For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status (kubernetes/kubernetes#125197, @gabesaba) [SIG Scheduling]

• K8s.io/apimachinery/pkg/util/runtime: new calls support handling panics and errors in the context where they occur. PanicHandlers and ErrorHandlers now must accept a context parameter for that. Log output is structured instead of unstructured. (kubernetes/kubernetes#121970, @pohly) [SIG API Machinery and Instrumentation]

• Kube-apiserver: the --encryption-provider-config file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. When --encryption-provider-config-automatic-reload is used, new encryption config files that contain typos after the kube-apiserver is running are treated as invalid and the last valid config is used. (kubernetes/kubernetes#124912, @enj) [SIG API Machinery and Auth]

• Kube-controller-manager removes deprecated command flags: --volume-host-cidr-denylist and --volume-host-allow-local-loopback (kubernetes/kubernetes#124017, @carlory) [SIG API Machinery, Apps, Cloud Provider and Storage]

• Kube-controller-manager: the horizontal-pod-autoscaler-upscale-delay and horizontal-pod-autoscaler-downscale-delay flags have been removed (deprecated and non-functional since v1.12) (kubernetes/kubernetes#124948, @SataQiu) [SIG API Machinery, Apps and Autoscaling]

• Support fine-grained supplemental groups policy (KEP-3619), which enables fine-grained control for supplementary groups in the first container processes. You can choose whether to include groups defined in the container image(/etc/groups) for the container's primary uid or not. (kubernetes/kubernetes#117842, @everpeace) [SIG API Machinery, Apps and Node]

• The kube-proxy nodeportAddresses / --nodeport-addresses option now
  accepts the value "primary", meaning to only listen for NodePort connections
  on the node's primary IPv4 and/or IPv6 address (according to the Node object).
  This is strongly recommended, if you were not previously using
  --nodeport-addresses, to avoid surprising behavior.

  (This behavior is enabled by default with the nftables backend; you would
  need to explicitly request --nodeport-addresses 0.0.0.0/0,::/0 there to get
  the traditional "listen on all interfaces" behavior.) (kubernetes/kubernetes#123105, @danwinship) [SIG API Machinery, Network and Windows]

v30.1.0

Kubernetes API Version: v1.30.1

New Feature:

• Add utility functions to parse and format GEP-2257 Duration strings for Gateway API

v30.1.0b1

Kubernetes API Version: v1.30.1

v30.1.0a1

Kubernetes API Version: v1.30.1

API Change Copy Section

• Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. (kubernetes/kubernetes#124553, @pmalek) [SIG API Machinery]

• Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. (kubernetes/kubernetes#124694, @pmalek) [SIG API Machinery]

• Added (alpha) support for the managedBy field on Jobs. Jobs with a custom value of this field - any value other than kubernetes.io/job-controller - were skipped by the job controller, and their reconciliation was delegated to an external controller, indicated by the value of the field. Jobs that didn't have this field at all, or where the field value was the reserved string kubernetes.io/job-controller, were reconciled by the built-in job controller.
  (kubernetes/kubernetes#123273, @mimowo)

• Added alpha-level support for the SuccessPolicy in Jobs.
  (kubernetes/kubernetes#123412, @tenzen-y)

• Added the CEL library for IP Addresses and CIDRs. This was made available for use starting from version 1.31.
  (kubernetes/kubernetes#121912, @JoelSpeed)

• Allowed container runtimes to fix an image garbage collection bug by adding an image_id field to the CRI Container message.
  (kubernetes/kubernetes#123508, @saschagrunert)

• Dynamic Resource Allocation: DRA drivers can now use "structured parameters" to let the scheduler handle claim allocation.
  (kubernetes/kubernetes#123516, @pohly)

• Fixed accidental enablement of the new alpha optionalOldSelf API field in CustomResourceDefinition validation rules, which should only have been allowed to be set when the CRDValidationRatcheting feature gate is enabled.
  (kubernetes/kubernetes#122329, @jpbetz)

• Implemented the prescore extension point for the volumeBinding plugin. It now returns skip if it doesn't do anything in Score.
  (kubernetes/kubernetes#115768, @AxeZhan)

• Kubelet would fail if NodeSwap was used with LimitedSwap and cgroupv1 node.
  (kubernetes/kubernetes#123738, @kannon92)

• Promoted AdmissionWebhookMatchConditions to GA. The feature is now stable, and the feature gate is now locked to default.
  (kubernetes/kubernetes#123560, @ivelichkovich)

• Structured Authentication Configuration now supports DiscoveryURL. If specified, discoveryURL overrides the URL used to fetch discovery information. This is for scenarios where the well-known and jwks endpoints are hosted at a different location than the issuer (such as locally in the cluster).
  (kubernetes/kubernetes#123527, @aramase)

• The StorageVersionMigration API, previously available as a Custom Resource Definition (CRD), is now a built-in API in Kubernetes.
  (kubernetes/kubernetes#123344, @nilekhc)

• When configuring a JWT authenticator:

  If username.expression used 'claims.email', then 'claims.email_verified' must have been used in username.expression or extra[*].valueExpression or claimValidationRules[*].expression. An example claim validation rule expression that matches the validation automatically applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true)'.
  (kubernetes/kubernetes#123737, @enj)

• readOnly volumes now support recursive read-only mounts for kernel versions >= 5.12."
  (kubernetes/kubernetes#123180, @AkihiroSuda)

• cri-api: Implemented KEP-3857: Recursive Read-only (RRO) mounts.
  (kubernetes/kubernetes#123272, @AkihiroSuda)

• kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1beta1.
  (kubernetes/kubernetes#123696, @aramase)

• kubelet allowed specifying a custom root directory for pod logs (instead of the default /var/log/pods) using the podLogsDir key in kubelet configuration.
  (kubernetes/kubernetes#112957, @mxpv)

• resource.k8s.io/ResourceClaim (alpha API): The strategic merge patch strategy for the status.reservedFor array was changed so that a strategic-merge-patch can now add individual entries. This change may break clients using strategic merge patch to update status, which rely on the previous behavior (replacing the entire array).
  (kubernetes/kubernetes#122276, @pohly)

• Added a CBOR implementation of runtime.Serializer. Until CBOR graduates to Alpha, API servers will refuse to start if configured with CBOR support. (kubernetes/kubernetes#122881, @benluddy)

• Added a alpha feature, behind the RelaxedEnvironmentVariableValidation feature gate.
  When that gate is enabled, Kubernetes allows almost all printable ASCII characters to be used in the names
  of environment variables for containers in Pods. (kubernetes/kubernetes#123385, @HirazawaUi)

• Added a new (alpha) field, trafficDistribution, to the Service spec to express preferences for traffic distribution to endpoints. Enabled through the ServiceTrafficDistribution feature gate. (kubernetes/kubernetes#123487, @gauravkghildiyal)

• Added audienceMatchPolicy field to AuthenticationConfiguration and support for configuring multiple audiences.
  The "audienceMatchPolicy" can be empty (or unset) when a single audience is specified in the "audiences" field.
  The "audienceMatchPolicy" must be set to "MatchAny" when multiple audiences are specified in the "audiences" field. (kubernetes/kubernetes#123165, @aramase)

• Added consistent vanity import to files and provided tooling for verifying and updating them. (kubernetes/kubernetes#120642, @jcchavezs)

• Added the disable-force-detach CLI option for kube-controller-manager. By default, it's set to false. When enabled, it prevents force detaching volumes based on maximum unmount time and node status. If activated, the non-graceful node shutdown feature must be used to recover from node failure. Additionally, if a pod needs to be forcibly terminated at the risk of corruption, the appropriate VolumeAttachment object must be deleted. (kubernetes/kubernetes#120344, @rohitssingh)

• Added to MutableFeatureGate the ability to override the default setting of feature gates, to allow default-enabling a feature on a component-by-component basis instead of for all affected components simultaneously. (kubernetes/kubernetes#122647, @benluddy)

• Aggregated discovery supports both v2beta1 and v2 types and feature is promoted to GA. (kubernetes/kubernetes#122882, @Jefftree)

• Alpha support for field selectors on custom resources has been added. With the CustomResourceFieldSelectors feature gate enabled, the CustomResourceDefinition API now allows specifying selectableFields. Listing a field there enables filtering custom resources for that CustomResourceDefinition in list or watch requests. (kubernetes/kubernetes#122717, @jpbetz)

• AppArmor profiles can now be configured through fields on the PodSecurityContext and container SecurityContext. The beta AppArmor annotations are deprecated, and AppArmor status is no longer included in the node ready condition. (kubernetes/kubernetes#123435, @tallclair)

• Contextual logging is now in beta and enabled by default. Check out the KEP and official documentation for more details. (kubernetes/kubernetes#122589, @pohly)

• Enabled concurrent log rotation in kubelet. You can now configure the maximum number of concurrent rotations with the containerLogMaxWorkers setting, and adjust the monitoring interval with containerLogMonitorInterval. (kubernetes/kubernetes#114301, @harshanarayana)

• Graduated pod scheduling gates to general availability.
  The PodSchedulingReadiness feature gate no longer has any effect, and the
  .spec.schedulingGates field is always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123575, @Huang-Wei)

• Graduated support for minDomains in pod topology spread constraints, to general availability.
  The MinDomainsInPodTopologySpread feature gate no longer has any effect, and the field is
  always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123481, @sanposhiho)

• In kubelet configuration, the .memorySwap.swapBehavior field now accepts a new value NoSwap, which becomes the default if unspecified. The previously accepted UnlimitedSwap value has been dropped.
  (kubernetes/kubernetes#122745, @kannon92)

• Kube-apiserver: the AuthorizationConfiguration type accepted in --authorization-config files has been promoted to apiserver.config.k8s.io/v1beta1. (kubernetes/kubernetes#123640, @liggitt)

• OIDC authentication will now fail if the username asserted based on a CEL expression config is the empty string. Previously the request would be authenticated with the username set to the empty string. (kubernetes/kubernetes#123568, @enj)

• Removed note that hostAliases are not supported on hostNetwork Pods from the PodSpec API. The feature has been supported since v1.8. (kubernetes/kubernetes#122422, @neolit123)

• Structured Authentication Configuration now supports configuring multiple JWT authenticators. The maximum allowed JWT authenticators in the authentication configuration is 64. (kubernetes/kubernetes#123431, @aramase)

• Text logging in Kubernetes components now uses textlogger. The same split streams of info and error log entries with buffering of info entries is now also supported for text output (off by default, alpha feature). Previously, this was only supported for JSON. Performance is better also without split streams. (kubernetes/kubernetes#114672, @pohly)

• The API server now detects and fails on startup if there are conflicting issuers between JWT authenticators and service account configurations. Previously, such configurations would run but could be inconsistently effective depending on the credential. (kubernetes/kubernetes#123561, @enj)

• The JWT authenticator configuration set via the --authentication-config flag is now dynamically reloaded as the file changes on disk. (kubernetes/kubernetes#123525, @enj)

• The StructuredAuthenticationConfiguration feature is now beta and enabled. (kubernetes/kubernetes#123719, @enj)

• The kube_codegen tool now ignores the vendor folder during code generation.
  (kubernetes/kubernetes#122729, @jparrill)

• The kubernetes repo now uses Go workspaces. This should not impact end users at all, but does have impact for developers of downstream projects. Switching to workspaces caused some breaking changes in the flags to the various k8s.io/code-generator tools. Downstream consumers should look at staging/src/k8s.io/code-generator/kube_codegen.sh to see the changes. (kubernetes/kubernetes#123529, @thockin)

• Updated an audit annotation key used by the …/serviceaccounts/<name>/token resource handler.
  The annotation used to persist the issued credential identifier is now authentication.kubernetes.io/issued-credential-id. (kubernetes/kubernetes#123098, @munnerz) [SIG Auth]

• Users are now allowed to mutate FSGroupPolicy and PodInfoOnMount in CSIDriver.Spec. (kubernetes/kubernetes#116209, @haoruan)

• ValidatingAdmissionPolicy was promoted to GA and will be enabled by default. (kubernetes/kubernetes#123405, @cici37)

• When scheduling a mix of pods using ResourceClaims and others that don't, scheduling a pod with ResourceClaims has a lower impact on scheduling latency. (kubernetes/kubernetes#121876, @pohly)

• When working with client-go events, it's now recommended to use NewEventBroadcasterAdapterWithContext instead of NewEventBroadcasterAdapter if contextual logging support is needed. (kubernetes/kubernetes#122142, @pohly)

• A new (alpha) field, trafficDistribution, has been added to the Service spec.
  This field provides a way to express preferences for how traffic is distributed to the endpoints for a Service.
  It can be enabled through the ServiceTrafficDistribution feature gate. (kubernetes/kubernetes#123487, @gauravkghildiyal) [SIG API Machinery, Apps and Network]

• Add alpha-level support for the SuccessPolicy in Jobs (kubernetes/kubernetes#123412, @tenzen-y) [SIG API Machinery, Apps and Testing]

• Added (alpha) support for the managedBy field on Jobs. Jobs with a custom value of this field - any
  value other than kubernetes.io/job-controller - are skipped by the job controller, and their
  reconciliation is delegated to an external controller, indicated by the value of the field. Jobs that
  don't have this field at all, or where the field value is the reserved string kubernetes.io/job-controller,
  are reconciled by the built-in job controller. (kubernetes/kubernetes#123273, @mimowo) [SIG API Machinery, Apps and Testing]

• Added a alpha feature, behind the RelaxedEnvironmentVariableValidation feature gate.
  When that gate is enabled, Kubernetes allows almost all printable ASCII characters to be used in the names
  of environment variables for containers in Pods. (kubernetes/kubernetes#123385, @HirazawaUi) [SIG Apps, Node and Testing]

• Added alpha support for field selectors on custom resources.
  Provided that the CustomResourceFieldSelectors feature gate is enabled, the CustomResourceDefinition
  API now lets you specify selectableFields. Listing a field there allows filtering custom resources for that
  CustomResourceDefinition in list or watch requests. (kubernetes/kubernetes#122717, @jpbetz) [SIG API Machinery]

• Added support for configuring multiple JWT authenticators in Structured Authentication Configuration. The maximum allowed JWT authenticators in the authentication configuration is 64. (kubernetes/kubernetes#123431, @aramase) [SIG Auth and Testing]

• Aggregated discovery supports both v2beta1 and v2 types and feature is promoted to GA (kubernetes/kubernetes#122882, @Jefftree) [SIG API Machinery and Testing]

• Allowing container runtimes to fix an image garbage collection bug by adding an image_id field to the CRI Container message. (kubernetes/kubernetes#123508, @saschagrunert) [SIG Node]

• AppArmor profiles can now be configured through fields on the PodSecurityContext and container SecurityContext.

  • The beta AppArmor annotations are deprecated.
  • AppArmor status is no longer included in the node ready condition (kubernetes/kubernetes#123435, @tallclair) [SIG API Machinery, Apps, Auth, Node and Testing]

• Conflicting issuers between JWT authenticators and service account config are now detected and fail on API server startup. Previously such a config would run but would be inconsistently effective depending on the credential. (kubernetes/kubernetes#123561, @enj) [SIG API Machinery and Auth]

• Dynamic Resource Allocation: DRA drivers may now use "structured parameters" to let the scheduler handle claim allocation. (kubernetes/kubernetes#123516, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Instrumentation, Node, Release, Scheduling, Storage and Testing]

• Graduated pod scheduling gates to general availability.
  The PodSchedulingReadiness feature gate no longer has any effect, and the
  .spec.schedulingGates field is always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123575, @Huang-Wei) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Graduated support for minDomains in pod topology spread constraints, to general availability.
  The MinDomainsInPodTopologySpread feature gate no longer has any effect, and the field is
  always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123481, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing]

• JWT authenticator config set via the --authentication-config flag is now dynamically reloaded as the file changes on disk. (kubernetes/kubernetes#123525, @enj) [SIG API Machinery, Auth and Testing]

• Kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1beta1. (kubernetes/kubernetes#123696, @aramase) [SIG API Machinery, Auth and Testing]

• Kube-apiserver: the AuthorizationConfiguration type accepted in --authorization-config files has been promoted to apiserver.config.k8s.io/v1beta1. (kubernetes/kubernetes#123640, @liggitt) [SIG Auth and Testing]

• Kubelet should fail if NodeSwap is used with LimitedSwap and cgroupv1 node. (kubernetes/kubernetes#123738, @kannon92) [SIG API Machinery, Node and Testing]

• Kubelet: a custom root directory for pod logs (instead of default /var/log/pods) can be specified using the podLogsDir
  key in kubelet configuration. (kubernetes/kubernetes#112957, @mxpv) [SIG API Machinery, Node, Scalability and Testing]

• Kubelet: the .memorySwap.swapBehavior field in kubelet configuration accepts a new value NoSwap and makes this the default if unspecified; the previously accepted UnlimitedSwap value has been dropped. (kubernetes/kubernetes#122745, @kannon92) [SIG API Machinery, Node and Testing]

• OIDC authentication will now fail if the username asserted based on a CEL expression config is the empty string. Previously the request would be authenticated with the username set to the empty string. (kubernetes/kubernetes#123568, @enj) [SIG API Machinery, Auth and Testing]

• PodSpec API: remove note that hostAliases are not supported on hostNetwork Pods. The feature has been supported since v1.8. (kubernetes/kubernetes#122422, @neolit123) [SIG API Machinery and Apps]

• Promote AdmissionWebhookMatchConditions to GA. The feature is now stable and the feature gate is now locked to default. (kubernetes/kubernetes#123560, @ivelichkovich) [SIG API Machinery and Testing]

• Structured Authentication Configuration now supports DiscoveryURL.
  discoveryURL if specified, overrides the URL used to fetch discovery information.
  This is for scenarios where the well-known and jwks endpoints are hosted at a different
  location than the issuer (such as locally in the cluster). (kubernetes/kubernetes#123527, @aramase) [SIG API Machinery, Auth and Testing]

• Support Recursive Read-only (RRO) mounts (KEP-3857) (kubernetes/kubernetes#123180, @AkihiroSuda) [SIG API Machinery, Apps, Node and Testing]

• The StructuredAuthenticationConfiguration feature is now beta and enabled by default. (kubernetes/kubernetes#123719, @enj) [SIG API Machinery and Auth]

• The StorageVersionMigration API, which was previously available as a Custom Resource Definition (CRD), is now a built-in API in Kubernetes. (kubernetes/kubernetes#123344, @nilekhc) [SIG API Machinery, Apps, Auth, CLI and Testing]

• The kubernetes repo now uses Go workspaces. This should not impact end users at all, but does have impact for developers of downstream projects. Switching to workspaces caused some breaking changes in the flags to the various k8s.io/code-generator tools. Downstream consumers should look at staging/src/k8s.io/code-generator/kube_codegen.sh to see the changes. (kubernetes/kubernetes#123529, @thockin) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing]

• ValidatingAdmissionPolicy is promoted to GA and will be enabled by default. (kubernetes/kubernetes#123405, @cici37) [SIG API Machinery, Apps, Auth and Testing]

• When configuring a JWT authenticator:

  If username.expression uses 'claims.email', then 'claims.email_verified' must be used in
  username.expression or extra[].valueExpression or claimValidationRules[].expression.
  An example claim validation rule expression that matches the validation automatically
  applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true)'. (kubernetes/kubernetes#123737, @enj) [SIG API Machinery and Auth]

• Added a CBOR implementation of runtime.Serializer. Until CBOR graduates to Alpha, API servers will refuse to start if configured with CBOR support. (kubernetes/kubernetes#122881, @benluddy) [SIG API Machinery]

• Added audienceMatchPolicy field to AuthenticationConfiguration and support for configuring multiple audiences.

  • The "audienceMatchPolicy" can be empty (or unset) when a single audience is specified in the "audiences" field.
  • The "audienceMatchPolicy" must be set to "MatchAny" when multiple audiences are specified in the "audiences" field. (kubernetes/kubernetes#123165, @aramase) [SIG API Machinery, Auth and Testing]

• Contextual logging is now beta and enabled by default. (kubernetes/kubernetes#122589, @pohly) [SIG Instrumentation]

• Cri-api: KEP-3857: Recursive Read-only (RRO) mounts (kubernetes/kubernetes#123272, @AkihiroSuda) [SIG Node]

• Enabled a mechanism for concurrent log rotatation via kubelet using a configuration entity of containerLogMaxWorkers which controls the maximum number of concurrent rotation that can be performed and an interval configuration of containerLogMonitorInterval that can aid is configuring the monitoring duration to best suite your cluster's log generation standards. (kubernetes/kubernetes#114301, @harshanarayana) [SIG API Machinery, Node and Testing]

• Text logging in Kubernetes components now uses textlogger. The same split streams of info and error log entries with buffering of info entries is now also supported for text output (off by default, alpha feature). Previously, this was only supported for JSON. Performance is better also without split streams. (kubernetes/kubernetes#114672, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]

• This change adds the following CLI option for kube-controller-manager:

  • disable-force-detach (defaults to false): Prevent force detaching volumes based on maximum unmount time and node status. If enabled, the non-graceful node shutdown feature must be used to recover from node failure (see https://kubernetes.io/blog/2023/08/16/kubernetes-1-28-non-graceful-node-shutdown-ga/). If enabled and a pod must be forcibly terminated at the risk of corruption, then the appropriate VolumeAttachment object (see here: https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume-attachment-v1/) must be deleted. (kubernetes/kubernetes#120344, @rohitssingh) [SIG API Machinery, Apps, Storage and Testing]

• Updated an audit annotation key used by the …/serviceaccounts/<name>/token resource handler.
  The annotation used to persist the issued credential identifier is now authentication.kubernetes.io/issued-credential-id. (kubernetes/kubernetes#123098, @munnerz) [SIG Auth]

• Add CEL library for IP Addresses and CIDRs. This will not be available for use until 1.31. (kubernetes/kubernetes#121912, @JoelSpeed) [SIG API Machinery]

• Added to MutableFeatureGate the ability to override the default setting of feature gates, to allow default-enabling a feature on a component-by-component basis instead of for all affected components simultaneously. (kubernetes/kubernetes#122647, @benluddy) [SIG API Machinery and Cluster Lifecycle]

• Adds a rule on the kube_codegen tool to ignore vendor folder during the code generation. (kubernetes/kubernetes#122729, @jparrill) [SIG API Machinery and Cluster Lifecycle]

• Allow users to mutate FSGroupPolicy and PodInfoOnMount in CSIDriver.Spec (kubernetes/kubernetes#116209, @haoruan) [SIG API Machinery, Storage and Testing]

• Client-go events: NewEventBroadcasterAdapterWithContext should be used instead of NewEventBroadcasterAdapter if the goal is to support contextual logging. (kubernetes/kubernetes#122142, @pohly) [SIG API Machinery, Instrumentation and Scheduling]

• Fixes accidental enablement of the new alpha optionalOldSelf API field in CustomResourceDefinition validation rules, which should only be allowed to be set when the CRDValidationRatcheting feature gate is enabled. (kubernetes/kubernetes#122329, @jpbetz) [SIG API Machinery]

• Implement prescore extension point for volumeBinding plugin. Return skip if it doesn't do anything in Score. (kubernetes/kubernetes#115768, @AxeZhan) [SIG Scheduling, Storage and Testing]

• Resource.k8s.io/ResourceClaim (alpha API): the strategic merge patch strategy for the status.reservedFor array was changed such that a strategic-merge-patch can add individual entries. This breaks clients using strategic merge patch to update status which rely on the previous behavior (replacing the entire array). (kubernetes/kubernetes#122276, @pohly) [SIG API Machinery]

• When scheduling a mixture of pods using ResourceClaims and others which don't, scheduling a pod with ResourceClaims impacts scheduling latency less. (kubernetes/kubernetes#121876, @pohly) [SIG API Machinery, Node, Scheduling and Testing]

v29.0.0

Kubernetes API Version: v1.29.0

Bug or Regression Copy Section

• Fix UTF-8 failures in Watch (#2100, @davidopic)
• Fix upper version boundary of urllib3, since other dependencies don't support urllib3 in version 2 (#2105, @jsaalfeld)

v29.0.0b1

Kubernetes API Version: v1.29.0

Bug or Regression Copy Section

• Fix UTF-8 failures in Watch (#2100, @davidopic)
• Fix upper version boundary of urllib3, since other dependencies don't support urllib3 in version 2 (#2105, @jsaalfeld)

v29.0.0a1

Kubernetes API Version: v1.29.0

API Change Copy Section

• 'kube-apiserver: adds --authentication-config flag for reading AuthenticationConfiguration
  files. --authentication-config flag is mutually exclusive with the existing --oidc-*
  flags.' (kubernetes/kubernetes#119142, @aramase)

• 'kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3
  is removed in v1.29. Migrated kube-scheduler configuration files to kubescheduler.config.k8s.io/v1.' (kubernetes/kubernetes#119994, @SataQiu)

• A new sleep action for the PreStop lifecycle hook was added, allowing containers to pause for a specified duration before termination. (kubernetes/kubernetes#119026, @AxeZhan)

• Added CEL expressions to v1alpha1 AuthenticationConfiguration. (kubernetes/kubernetes#121078, @aramase)

• Added Windows support for InPlace Pod Vertical Scaling feature. (kubernetes/kubernetes#112599, @fabi200123) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]

• Added ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. (kubernetes/kubernetes#121275, @haircommander)

• Added UserNamespacesPodSecurityStandards feature gate to enable user namespace support for Pod Security Standards.
  Enabling this feature will modify all Pod Security Standard rules to allow setting: spec[.*].securityContext.[runAsNonRoot,runAsUser].
  This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
  The feature gate will not graduate or be enabled by default in future Kubernetes releases. (kubernetes/kubernetes#118760, @saschagrunert) [SIG API Machinery, Auth, Node and Release]

• Added optionalOldSelf to x-kubernetes-validations to support ratcheting CRD schema constraints. (kubernetes/kubernetes#121034, @alexzielenski)

• Added a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses. (kubernetes/kubernetes#116516, @aojea)

• Added a new ipMode field to the .status of Services where type is set to LoadBalancer.
  The new field is behind the LoadBalancerIPMode feature gate. (kubernetes/kubernetes#119937, @RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]

• Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. (kubernetes/kubernetes#120808, @aroradaman)

• Added support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. (kubernetes/kubernetes#121223, @ritazh)

• Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. (kubernetes/kubernetes#113374, @ahmedtd)

• Added the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. (kubernetes/kubernetes#120954, @HirazawaUi)

• Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
  The incorrect cost was evident when the result of a function was used in subsequent operations. (kubernetes/kubernetes#119800, @jpbetz) [SIG API Machinery, Auth and Cloud Provider]

• Fixed the API comments for the Job Ready field in status. (kubernetes/kubernetes#121765, @mimowo)

• Fixed the API comments for the FailIndex Job pod failure policy action. (kubernetes/kubernetes#121764, @mimowo)

• Go API: the ResourceRequirements struct was replaced with VolumeResourceRequirements for use with volumes. (kubernetes/kubernetes#118653, @pohly)

• Graduated Job BackoffLimitPerIndex feature to beta. (kubernetes/kubernetes#121356, @mimowo)

• Marked the onPodConditions field as optional in Job's pod failure policy. (kubernetes/kubernetes#120204, @mimowo)

• Promoted PodReadyToStartContainers condition to beta. (kubernetes/kubernetes#119659, @kannon92)

• The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes:

  • PriorityLevelConfiguration: the .spec.limited.nominalConcurrencyShares field defaults to 30 only if the field is omitted (v1beta3 also defaulted an explicit 0 value to 30). Specifying an explicit 0 value is not allowed in the v1 version in v1.29 to ensure compatibility with v1.28 API servers. In v1.30, explicit 0 values will be allowed in this field in the v1 API.
    The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the v1 APIs. Transition clients and manifests to use the v1 APIs before upgrading to v1.32. (kubernetes/kubernetes#121089, @tkashem)

• The kube-proxy command-line documentation was updated to clarify that
  --bind-address does not actually have anything to do with binding to an
  address, and you probably don't actually want to be using it. (kubernetes/kubernetes#120274, @danwinship)

• The kube-scheduler selectorSpread plugin has been removed, please use the podTopologySpread plugin instead. (kubernetes/kubernetes#117720, @kerthcet)

• The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. (kubernetes/kubernetes#116065, @sanposhiho)

• When updating a CRD, per-expression cost limit check are now skipped for x-kubernetes-validations rules of versions that are not mutated. (kubernetes/kubernetes#121460, @jiahuif)

• CSINodeExpandSecret feature has been promoted to GA in this release and is enabled
  by default. The CSI drivers can make use of the secretRef values passed in NodeExpansion
  request optionally sent by the CSI Client from this release onwards. (kubernetes/kubernetes#121303, @humblec)

• NodeStageVolume calls will now be retried if the CSI node driver is not running. (kubernetes/kubernetes#120330, @rohitssingh)

• PersistentVolumeLastPhaseTransitionTime is now beta and enabled by default. (kubernetes/kubernetes#120627, @RomanBednar)

• ValidatingAdmissionPolicy type checking now supports CRDs and API extensions types. (kubernetes/kubernetes#119109, @jiahuif)

• kube-apiserver: added --authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. The --authorization-config flag is mutually exclusive with --authorization-modes and --authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for --authorization-config to be specified. (kubernetes/kubernetes#120154, @palnabarun)

• kube-proxy now has a new nftables-based mode, available by running

  text Copy

  `kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables`

  This is currently an alpha-level feature and while it probably will not
  eat your data, it may nibble at it a bit. (It passes e2e testing but has
  not yet seen real-world use.)

  At this point it should be functionally mostly identical to the iptables
  mode, except that it does not (and will not) support Service NodePorts on
  127.0.0.1. (Also note that there are currently no command-line arguments
  for the nftables-specific config; you will need to use a config file if
  you want to set the equivalent of any of the --iptables-xxx options.)

  As this code is still very new, it has not been heavily optimized yet;
  while it is expected to eventually have better performance than the
  iptables backend, very little performance testing has been done so far. (kubernetes/kubernetes#121046, @danwinship)

• kube-proxy: Added an option/flag for configuring the nf_conntrack_tcp_be_liberal sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. (kubernetes/kubernetes#120354, @aroradaman)

• Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. (kubernetes/kubernetes#113374, @ahmedtd) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]

• Adds optionalOldSelf to x-kubernetes-validations to support ratcheting CRD schema constraints (kubernetes/kubernetes#121034, @alexzielenski) [SIG API Machinery]

• Fix API comment for the Job Ready field in status (kubernetes/kubernetes#121765, @mimowo) [SIG API Machinery and Apps]

• Fix API comments for the FailIndex Job pod failure policy action. (kubernetes/kubernetes#121764, @mimowo) [SIG API Machinery and Apps]

• A new sleep action for the PreStop lifecycle hook is added, allowing containers to pause for a specified duration before termination. (kubernetes/kubernetes#119026, @AxeZhan) [SIG API Machinery, Apps, Node and Testing]

• Add ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. (kubernetes/kubernetes#121275, @haircommander) [SIG API Machinery and Node]

• Add a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses (kubernetes/kubernetes#116516, @aojea) [SIG API Machinery, Apps, Auth, CLI, Network and Testing]

• Add the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. (kubernetes/kubernetes#120954, @HirazawaUi) [SIG API Machinery, Apps and Node]

• Added Windows support for InPlace Pod Vertical Scaling feature. (kubernetes/kubernetes#112599, @fabi200123) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]

• Added UserNamespacesPodSecurityStandards feature gate to enable user namespace support for Pod Security Standards.
  Enabling this feature will modify all Pod Security Standard rules to allow setting: spec[.*].securityContext.[runAsNonRoot,runAsUser].
  This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
  The feature gate will not graduate or be enabled by default in future Kubernetes releases. (kubernetes/kubernetes#118760, @saschagrunert) [SIG API Machinery, Auth, Node and Release]

• Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. (kubernetes/kubernetes#120808, @aroradaman) [SIG API Machinery and Network]

• Adds CEL expressions to v1alpha1 AuthenticationConfiguration. (kubernetes/kubernetes#121078, @aramase) [SIG API Machinery, Auth and Testing]

• Adds support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. (kubernetes/kubernetes#121223, @ritazh) [SIG API Machinery and Auth]

• CSINodeExpandSecret feature has been promoted to GA in this release and enabled by default. The CSI drivers can make use of the secretRef values passed in NodeExpansion request optionally sent by the CSI Client from this release onwards. (kubernetes/kubernetes#121303, @humblec) [SIG API Machinery, Apps and Storage]

• Graduate Job BackoffLimitPerIndex feature to Beta (kubernetes/kubernetes#121356, @mimowo) [SIG Apps]

• Kube-apiserver: adds --authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. --authorization-config flag is mutually exclusive with --authorization-modes and --authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for --authorization-config to be specified. (kubernetes/kubernetes#120154, @palnabarun) [SIG API Machinery, Auth and Testing]

• Kube-proxy now has a new nftables-based mode, available by running

  text Copy

  kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables

  This is currently an alpha-level feature and while it probably will not
  eat your data, it may nibble at it a bit. (It passes e2e testing but has
  not yet seen real-world use.)

  At this point it should be functionally mostly identical to the iptables
  mode, except that it does not (and will not) support Service NodePorts on
  127.0.0.1. (Also note that there are currently no command-line arguments
  for the nftables-specific config; you will need to use a config file if
  you want to set the equivalent of any of the --iptables-xxx options.)

  As this code is still very new, it has not been heavily optimized yet;
  while it is expected to eventually have better performance than the
  iptables backend, very little performance testing has been done so far. (kubernetes/kubernetes#121046, @danwinship) [SIG API Machinery and Network]

• Kube-proxy: Added an option/flag for configuring the nf_conntrack_tcp_be_liberal sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. (kubernetes/kubernetes#120354, @aroradaman) [SIG API Machinery and Network]

• PersistentVolumeLastPhaseTransitionTime is now beta, enabled by default. (kubernetes/kubernetes#120627, @RomanBednar) [SIG Storage]

• Promote PodReadyToStartContainers condition to beta. (kubernetes/kubernetes#119659, @kannon92) [SIG Node and Testing]

• The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes:

  • PriorityLevelConfiguration: the .spec.limited.nominalConcurrencyShares field defaults to 30 only if the field is omitted (v1beta3 also defaulted an explicit 0 value to 30). Specifying an explicit 0 value is not allowed in the v1 version in v1.29 to ensure compatibility with 1.28 API servers. In v1.30, explicit 0 values will be allowed in this field in the v1 API.
    The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the v1 APIs. Transition clients and manifests to use the v1 APIs before upgrading to v1.32. (kubernetes/kubernetes#121089, @tkashem) [SIG API Machinery and Testing]

• The kube-proxy command-line documentation was updated to clarify that
  --bind-address does not actually have anything to do with binding to an
  address, and you probably don't actually want to be using it. (kubernetes/kubernetes#120274, @danwinship) [SIG Network]

• The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. (kubernetes/kubernetes#116065, @sanposhiho) [SIG API Machinery, Apps, Cloud Provider, Scheduling and Testing]

• ValidatingAdmissionPolicy Type Checking now supports CRDs and API extensions types. (kubernetes/kubernetes#119109, @jiahuif) [SIG API Machinery, Apps, Auth and Testing]

• When updating a CRD, per-expression cost limit check is skipped for x-kubernetes-validations rules of versions that are not mutated. (kubernetes/kubernetes#121460, @jiahuif) [SIG API Machinery]

• Added a new ipMode field to the .status of Services where type is set to LoadBalancer.
  The new field is behind the LoadBalancerIPMode feature gate. (kubernetes/kubernetes#119937, @RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]

• Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
  The incorrect cost was evident when the result of a function was used in subsequent operations. (kubernetes/kubernetes#119800, @jpbetz) [SIG API Machinery, Auth and Cloud Provider]

• Go API: the ResourceRequirements struct needs to be replaced with VolumeResourceRequirements for use with volumes. (kubernetes/kubernetes#118653, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing]

• Kube-apiserver: adds --authentication-config flag for reading AuthenticationConfiguration files. --authentication-config flag is mutually exclusive with the existing --oidc-* flags. (kubernetes/kubernetes#119142, @aramase) [SIG API Machinery, Auth and Testing]

• Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3 is removed in v1.29. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (kubernetes/kubernetes#119994, @SataQiu) [SIG Scheduling and Testing]

• Mark the onPodConditions field as optional in Job's pod failure policy. (kubernetes/kubernetes#120204, @mimowo) [SIG API Machinery and Apps]

• Retry NodeStageVolume calls if CSI node driver is not running (kubernetes/kubernetes#120330, @rohitssingh) [SIG Apps, Storage and Testing]

• The kube-scheduler selectorSpread plugin has been removed, please use the podTopologySpread plugin instead. (kubernetes/kubernetes#117720, @kerthcet) [SIG Scheduling]

v28.1.0

Kubernetes API Version: v1.28.2

API Change Copy Section

• Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
  The incorrect cost was evident when the result of a function was used in subsequent operations. (kubernetes/kubernetes#119807, @jpbetz) [SIG API Machinery, Auth and Cloud Provider]
• Mark Job onPodConditions as optional in pod failure policy (kubernetes/kubernetes#120208, @mimowo) [SIG API Machinery and Apps]

v28.1.0b1

Kubernetes API Version: v1.28.1

v28.1.0a1

Kubernetes API Version: v1.28.1

API Change Copy Section

• A CDIDevice field is included in the Device Plugin's ContainerAllocateResponse. This field maps to the CDIDevice field in the CRI protocol. (kubernetes/kubernetes#118254, @elezar) [SIG Node and Testing]
• ACTION_REQUIRED
  When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. (kubernetes/kubernetes#118420, @alculquicondor) [SIG Apps]
• Added ServedVersions field to StorageVersion API. (kubernetes/kubernetes#118386, @Richabanker)
• Added IP mode field to loadbalancer status ingress. (kubernetes/kubernetes#118895, @RyanAoh)
• Added podReplacementPolicy and terminating field to job api. (kubernetes/kubernetes#119301, @kannon92)
• Added a new namespaceParamRef field to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy. (kubernetes/kubernetes#119215, @alexzielenski) [SIG API Machinery and Testing]
• Added a warning that TLS 1.3 ciphers are not configurable. (kubernetes/kubernetes#115399, @3u13r) [SIG API Machinery and Node]
• Added error handling for seccomp localhost configurations that do not properly set a localhostProfile. (kubernetes/kubernetes#117020, @cji)
• Added fields reason and fieldPath into CRD validation rules to allow users to specify reason and field path when validation failed. (kubernetes/kubernetes#118041, @cici37) [SIG API Machinery]
• Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a namespaceObject
  variable with expressions. (kubernetes/kubernetes#118267, @cici37) [SIG API Machinery and Testing]
• Added new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. (kubernetes/kubernetes#118990, @alexzielenski)
• Added new annotation batch.kubernetes.io/cronjob-scheduled-timestamp to Job objects scheduled from CronJobs. (kubernetes/kubernetes#118137, @helayoty)
• Added new config option delayCacheUntilActive to KubeSchedulerConfiguration that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in kube-scheduler (kubernetes/kubernetes#115754, @linxiulei) [SIG API Machinery and Scheduling]
• Changed how KMS v2 encryption at rest can generate data encryption keys.
  When you enable the KMSv2KDF feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. (kubernetes/kubernetes#118828, @enj)
• Exposed rest.DefaultServerUrlFor function. (kubernetes/kubernetes#118055, @timofurrer)
• Extended the Job API for alpha version of BackoffLimitPerIndex. (kubernetes/kubernetes#119294, @mimowo)
• Graduated AdmissionWebhookMatchCondition feature to beta. (kubernetes/kubernetes#119380, @a-hilaly)
• If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via memory.oom.group . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. (kubernetes/kubernetes#117793, @tzneal) [SIG Apps, Node and Testing]
• In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . (kubernetes/kubernetes#118782, @MikeSpreitzer) [SIG API Machinery]
• Indexed Job pods now have the pod completion index set as a pod label. (kubernetes/kubernetes#118883, @danielvegamyhre) [SIG Apps]
• Kube-proxy: added --logging-format flag to support structured logging. (kubernetes/kubernetes#117800, @cyclinder)
• NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. (kubernetes/kubernetes#115398, @tangwz) [SIG Scheduling]
• PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. (kubernetes/kubernetes#116469, @RomanBednar)
• Pods which set hostNetwork: true and declare ports, get the hostPort field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now hostPort will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. (kubernetes/kubernetes#117696, @thockin) [SIG Apps]
• Promoted API groups ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding to v1beta1. (kubernetes/kubernetes#118644, @alexzielenski) [SIG API Machinery, Apps and Testing]
• Promoted the feature gate ValidtaingAdmissionPolicy to beta, and it is turned off by default. (kubernetes/kubernetes#119409, @alexzielenski)
• Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to BETA stability. (kubernetes/kubernetes#119264, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
• Removed resizeStatus enum from pvc.Status and replaced with AllocatedResourceStatus. (kubernetes/kubernetes#116335, @gnufied) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
• Removed WindowsHostProcessContainers feature-gate. (kubernetes/kubernetes#117570, @marosset) [SIG API Machinery, Apps, Auth, Node and Windows]
• Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta. (kubernetes/kubernetes#117802, @kerthcet) [SIG API Machinery and Apps]
• StatefulSet pods now have the pod index set as a pod label statefulset.kubernetes.io/pod-index. (kubernetes/kubernetes#119232, @danielvegamyhre) [SIG Apps]
• Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver (kubernetes/kubernetes#117740, @Richabanker) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
• Supported BackoffLimitPerIndex in Jobs. (kubernetes/kubernetes#118009, @mimowo)
• The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
  creates the KUBE-MARK-DROP chain (which has been unused for several releases)
  or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). (kubernetes/kubernetes#119374, @danwinship)
• The SelfSubjectReview API is promoted to authentication.k8s.io/v1 and the kubectl auth whoami command is GA. (kubernetes/kubernetes#117713, @nabokihms) [SIG API Machinery, Architecture, Auth, CLI and Testing]
• The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still -<claim name>, but a random suffix will avoid name collisions. (kubernetes/kubernetes#117351, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
• The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. (kubernetes/kubernetes#116429, @gjkim42) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Updated the comment about the feature-gate level for PodFailurePolicy from alpha to beta (kubernetes/kubernetes#118278, @mimowo)
• client-go: Improved memory use of reflector caches when watching large numbers
  of objects which do not change frequently. (kubernetes/kubernetes#113362, @sxllwx)
• component-base/logs is now stricter about not applying configurations multiple
  times and will return an error when that is attempted. Can be overridden by binaries
  which need to do that. (kubernetes/kubernetes#117108, @pohly)
• kube-controller-manager: The LegacyServiceAccountTokenCleanUp feature gate
  is now available as alpha (off by default). When enabled, the legacy-service-account-token-cleaner
  controller loop removes service account token secrets that have not been used
  in the time specified by --legacy-service-account-token-clean-up-period (defaulting
  to one year), and are referenced from the .secrets list of a ServiceAccount
  object, and are not referenced from pods. (kubernetes/kubernetes#115554, @yt2985)
• kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2
  is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (kubernetes/kubernetes#117649, @SataQiu)
• Aggregated discovery now returns responseKind: {} for resources which are missing group/version/kind information, to ensure compatibility with v0.26.0-v0.26.3 clients. (kubernetes/kubernetes#119835, @liggitt) [SIG API Machinery and Testing]
• Fix CustomResourceDefinition status.storedVersions validation error messages. (kubernetes/kubernetes#119653, @sttts) [SIG API Machinery]
• Kube-proxy in Kubernetes >= 1.28 up until v1.28.0-beta.0 ignored the -v command line flag when combined with --config. (kubernetes/kubernetes#119867, @pohly) [SIG Network]
• PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. (kubernetes/kubernetes#116469, @RomanBednar) [SIG API Machinery, Apps, Auth, Node, Release, Storage and Testing]
• Promoted API groups ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding to v1beta1. (kubernetes/kubernetes#118644, @alexzielenski) [SIG API Machinery, Apps and Testing]
• Promoted the feature gate ValidtaingAdmissionPolicy to beta and it is turned off by default. (kubernetes/kubernetes#119409, @alexzielenski) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Release, Storage and Testing]
• Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the KMSv2KDF feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. (kubernetes/kubernetes#118828, @enj) [SIG API Machinery, Auth and Testing]
• A CDIDevice field is includes in the Device Plugin's ContainerAllocateResponse. This field maps to the CDIDevice field in the CRI protocol. (kubernetes/kubernetes#118254, @elezar) [SIG Node and Testing]
• Add new annotation batch.kubernetes.io/cronjob-scheduled-timestamp to Job objects scheduled from CronJobs. (kubernetes/kubernetes#118137, @helayoty) [SIG Apps]
• Add podReplacementPolicy and terminating field to job api (kubernetes/kubernetes#119301, @kannon92) [SIG API Machinery and Apps]
• Added fields reason and fieldPath into CRD validation rules to allow users to specify reason and field path when validation failed. (kubernetes/kubernetes#118041, @cici37) [SIG API Machinery]
• Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a namespaceObject
  variable with expressions. (kubernetes/kubernetes#118267, @cici37) [SIG API Machinery and Testing]
• Adds new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. (kubernetes/kubernetes#118990, @alexzielenski) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• Adds new namespaceParamRef to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy (kubernetes/kubernetes#119215, @alexzielenski) [SIG API Machinery and Testing]
• Extend the Job API for alpha version of BackoffLimitPerIndex (kubernetes/kubernetes#119294, @mimowo) [SIG API Machinery and Apps]
• Graduate AdmissionWebhookMatchCondition feature to beta (kubernetes/kubernetes#119380, @a-hilaly) [SIG API Machinery]
• In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . (kubernetes/kubernetes#118782, @MikeSpreitzer) [SIG API Machinery]
• Indexed Job pods now have the pod completion index set as a pod label. (kubernetes/kubernetes#118883, @danielvegamyhre) [SIG Apps]
• Kube-proxy: add '--logging-format' flag to support structured logging (kubernetes/kubernetes#117800, @cyclinder) [SIG API Machinery, Architecture, Instrumentation and Network]
• Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to BETA stability. (kubernetes/kubernetes#119264, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
• Removed resizeStatus enum from pvc.Status and replaced with AllocatedResourceStatus (kubernetes/kubernetes#116335, @gnufied) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
• StatefulSet pods now have the pod index set as a pod label statefulset.kubernetes.io/pod-index. (kubernetes/kubernetes#119232, @danielvegamyhre) [SIG Apps]
• Support BackoffLimitPerIndex in Jobs (kubernetes/kubernetes#118009, @mimowo) [SIG API Machinery, Apps and Testing]
• Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver (kubernetes/kubernetes#117740, @Richabanker) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
• The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
  creates the KUBE-MARK-DROP chain (which has been unused for several releases)
  or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). (kubernetes/kubernetes#119374, @danwinship) [SIG API Machinery, Network and Node]
• The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still -<claim name>, but a random suffix will avoid name collisions. (kubernetes/kubernetes#117351, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
• The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. (kubernetes/kubernetes#116429, @gjkim42) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Add ServedVersions field to StorageVersion API (kubernetes/kubernetes#118386, @Richabanker) [SIG API Machinery and Testing]
• Component-base/logs is now more strict about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. (kubernetes/kubernetes#117108, @pohly) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Scheduling and Testing]
• ACTION_REQUIRED
  When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. (kubernetes/kubernetes#118420, @alculquicondor) [SIG Apps]
• Expose rest.DefaultServerUrlFor function (kubernetes/kubernetes#118055, @timofurrer) [SIG API Machinery]
• If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via memory.oom.group . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. (kubernetes/kubernetes#117793, @tzneal) [SIG Apps, Node and Testing]
• Update the comment about the feature-gate level for PodFailurePolicy from alpha to beta (kubernetes/kubernetes#118278, @mimowo) [SIG Apps]
• Added a warning that TLS 1.3 ciphers are not configurable. (kubernetes/kubernetes#115399, @3u13r) [SIG API Machinery and Node]
• Added error handling for seccomp localhost configurations that do not properly set a localhostProfile (kubernetes/kubernetes#117020, @cji) [SIG API Machinery and Node]
• Added new config option delayCacheUntilActive to KubeSchedulerConfiguration that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in kube-scheduler (kubernetes/kubernetes#115754, @linxiulei) [SIG API Machinery and Scheduling]
• Client-go: Improved memory use of reflector caches when watching large numbers of objects which do not change frequently (kubernetes/kubernetes#113362, @sxllwx) [SIG API Machinery]
• Kube-controller-manager: The LegacyServiceAccountTokenCleanUp feature gate is now available as alpha (off by default). When enabled, the legacy-service-account-token-cleaner controller loop removes service account token secrets that have not been used in the time specified by --legacy-service-account-token-clean-up-period (defaulting to one year), and are referenced from the .secrets list of a ServiceAccount object, and are not referenced from pods. (kubernetes/kubernetes#115554, @yt2985) [SIG API Machinery, Apps, Auth, Release and Testing]
• Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2 is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (kubernetes/kubernetes#117649, @SataQiu) [SIG API Machinery, Scheduling and Testing]
• NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. (kubernetes/kubernetes#115398, @tangwz) [SIG Scheduling]
• Pods which set hostNetwork: true and declare ports get the hostPort field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now hostPort will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. (kubernetes/kubernetes#117696, @thockin) [SIG Apps]
• Removing WindowsHostProcessContainers feature-gate (kubernetes/kubernetes#117570, @marosset) [SIG API Machinery, Apps, Auth, Node and Windows]
• Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta (kubernetes/kubernetes#117802, @kerthcet) [SIG API Machinery and Apps]
• The SelfSubjectReview API is promoted to authentication.k8s.io/v1 and the kubectl auth whoami command is GA. (kubernetes/kubernetes#117713, @nabokihms) [SIG API Machinery, Architecture, Auth, CLI and Testing]

v27.2.0

Kubernetes API Version: v1.27.3

Documentation Copy Section

• Fix request_timeout example and doc. Arg name should be _request_timeout. Single value type should be int or long. (#2071, @hemslo)

v27.2.0b1

Kubernetes API Version: v1.27.2

v27.2.0a1

Kubernetes API Version: v1.27.2

API Change Copy Section

• Added error handling for seccomp localhost configurations that do not properly set a localhostProfile (kubernetes/kubernetes#117020, @cji) [SIG API Machinery and Node]

• Fixed an issue where kubelet does not set case-insensitive headers for http probes. (#117182, @dddddai) (kubernetes/kubernetes#117324, @dddddai) [SIG API Machinery, Apps and Node]

• Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta (kubernetes/kubernetes#117815, @kerthcet) [SIG Apps]

• A fix in the resource.k8s.io/v1alpha1/ResourceClaim API avoids harmless (?) ".status.reservedFor: element 0: associative list without keys has an element that's a map type" errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. (kubernetes/kubernetes#115354, @pohly)

• A terminating pod on a node that is not caused by preemption no longer prevents kube-scheduler from preempting pods on that node

  • Rename PreemptionByKubeScheduler to PreemptionByScheduler (kubernetes/kubernetes#114623, @Huang-Wei)

• API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. (kubernetes/kubernetes#116556, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]

• Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
  restrictions that already apply to CustomResourceDefinition.
  If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
  admission check that was being performed is aborted; the failurePolicy for the ValidatingAdmissionPolicy
  determines the outcome. (kubernetes/kubernetes#115747, @cici37)

• Added auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
  Added validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. (kubernetes/kubernetes#115973, @jpbetz)

• Added messageExpression field to ValidationRule. (kubernetes/kubernetes#115969, @DangerOnTheRanger)

• Added messageExpression to ValidatingAdmissionPolicy, to set custom failure message via CEL expression. (kubernetes/kubernetes#116397, @jiahuif) [SIG API Machinery]

• Added a new IPAddress object kind

  • Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 (kubernetes/kubernetes#115075, @aojea) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]

• Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
  A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (kubernetes/kubernetes#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]

• Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a authorizer
  variable with expressions. The new variable provides a builder that allows expressions such authorizer.group('').resource('pods').check('create').allowed(). (kubernetes/kubernetes#116054, @jpbetz) [SIG API Machinery and Testing]

• Added matchConditions field to ValidatingAdmissionPolicy and enabled support for CEL based custom match criteria. (kubernetes/kubernetes#116350, @maxsmythe)

• Added new option to the InterPodAffinity scheduler plugin to ignore existing
  podspreferred inter-pod affinities if the incoming pod has no preferred inter-pod affinities. This option can be used as an optimization for higher scheduling throughput (at the cost of an occasional pod being scheduled non-optimally/violating existing pods preferred inter-pod affinities). To enable this scheduler option, set theInterPodAffinityscheduler plugin argignorePreferredTermsOfExistingPods: true` (kubernetes/kubernetes#114393, @danielvegamyhre)

• Added the MatchConditions field to ValidatingWebhookConfiguration and MutatingWebhookConfiguration for the v1beta and v1 apis.

  The AdmissionWebhookMatchConditions featuregate is now in Alpha (kubernetes/kubernetes#116261, @ivelichkovich) [SIG API Machinery and Testing]

• Added validation to ensure that if service.kubernetes.io/topology-aware-hints and service.kubernetes.io/topology-mode annotations are both set, they are set to the same value.Also Added deprecation warning if service.kubernetes.io/topology-aware-hints annotation is used. (kubernetes/kubernetes#116612, @robscott)

• Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. (kubernetes/kubernetes#114412, @thockin)

• Adds feature gate NodeLogQuery which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. (kubernetes/kubernetes#96120, @LorbusChris)

• Api: validation of a PodSpec now rejects invalid ResourceClaim and ResourceClaimTemplate names. For a pod, the name generated for the ResourceClaim when using a template also must be valid. (kubernetes/kubernetes#116576, @pohly)

• Bump default API QPS limits for Kubelet. (kubernetes/kubernetes#116121, @wojtek-t)

• Enabled the StatefulSetStartOrdinal feature gate in beta (kubernetes/kubernetes#115260, @pwschuurman)

• Enabled usage of kube-proxy, kube-scheduler and kubelet HTTP APIs for changing the logging
  verbosity at runtime for JSON output. (kubernetes/kubernetes#114609, @pohly)

• Encryption of API Server at rest configuration now allows the use of wildcards in the list of resources. For example, . can be used to encrypt all resources, including all current and future custom resources. (kubernetes/kubernetes#115149, @nilekhc)

• Extended the kubelet's PodResources API to include resources allocated in ResourceClaims via DynamicResourceAllocation. Additionally, added a new Get() method to query a specific pod for its resources. (kubernetes/kubernetes#115847, @moshe010) [SIG Node]

• Forbid to set matchLabelKeys when labelSelector is not set in topologySpreadConstraints (kubernetes/kubernetes#116535, @denkensk)

• GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) (kubernetes/kubernetes#115966, @aojea) [SIG Apps and Cloud Provider]

• GRPC probes are now a GA feature. GRPCContainerProbe feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. (kubernetes/kubernetes#116233, @SergeyKanzhelev)

• Graduated Kubelet Topology Manager to GA. (kubernetes/kubernetes#116093, @swatisehgal)

• Graduated KubeletTracing to beta, which means that the feature gate is now enabled by default. (kubernetes/kubernetes#115750, @saschagrunert)

• Graduated seccomp profile defaulting to GA.

  Set the kubelet --seccomp-default flag or seccompDefault kubelet configuration field to true to make pods on that node default to using the RuntimeDefault seccomp profile.

  Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.

  Guidance for identifying and mitigating those issues is outlined in the Kubernetes seccomp tutorial. (kubernetes/kubernetes#115719, @saschagrunert) [SIG API Machinery, Node, Storage and Testing]

• Graduated the container resource metrics feature on HPA to beta. (kubernetes/kubernetes#116046, @sanposhiho)

• Implemented API streaming for the watch-cache

  When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (kubernetes/kubernetes#110960, @p0lyn0mial)

• Introduced API for streaming.

  Added SendInitialEvents field to the ListOptions. When the new option is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (kubernetes/kubernetes#115402, @p0lyn0mial)

• Introduced a breaking change to the resource.k8s.io API in its AllocationResult struct. This change allows a kubelet plugin for the DynamicResourceAllocation feature to service allocations from multiple resource driver controllers. (kubernetes/kubernetes#116332, @klueska)

• Introduces new alpha functionality to the reflector, allowing user to enable API streaming.

  To activate this feature, users can set the ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environmental variable.
  It is important to note that the server must support streaming for this feature to function properly.
  If streaming is not supported by the server, the reflector will revert to the previous method
  of obtaining data through LIST/WATCH semantics. (kubernetes/kubernetes#110772, @p0lyn0mial) [SIG API Machinery]

• K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. (kubernetes/kubernetes#115514, @pohly) [SIG API Machinery]

• K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message (kubernetes/kubernetes#114680, @pohly) [SIG Instrumentation]

• Kubeadm: explicitly set priority for static pods with priorityClassName: system-node-critical (kubernetes/kubernetes#114338, @champtar) [SIG Cluster Lifecycle]

• Kubelet: a "maxParallelImagePulls" field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. (kubernetes/kubernetes#115220, @ruiwen-zhao) [SIG API Machinery, Node and Scalability]

• Kubelet: changed MemoryThrottlingFactor default value to 0.9 and formulas to calculate memory.high (kubernetes/kubernetes#115371, @pacoxu)

• Kubernetes components that perform leader election now only support using Leases for this. (kubernetes/kubernetes#114055, @aimuz)

• Migrated the DaemonSet controller (within kube-controller-manager) to use contextual logging (kubernetes/kubernetes#113622, @249043822)

• New service.kubernetes.io/topology-mode annotation has been introduced as a replacement for the service.kubernetes.io/topology-aware-hints annotation.

  • service.kubernetes.io/topology-aware-hints annotation has been deprecated.
  • kube-proxy now accepts any value that is not "disabled" for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. (kubernetes/kubernetes#116522, @robscott) [SIG Apps, Network and Testing]

• Pods owned by a Job now uses the labels batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid.
  The legacy labels job-name and controller-uid are still added for compatibility. (kubernetes/kubernetes#114930, @kannon92)

• Promoted CronJobTimeZone feature to GA (kubernetes/kubernetes#115904, @soltysh)

• Promoted SelfSubjectReview to Beta (kubernetes/kubernetes#116274, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]

• Relaxed API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). (kubernetes/kubernetes#116161, @danielvegamyhre)

• Remove kubernetes.io/grpc standard appProtocol (kubernetes/kubernetes#116866, @LiorLieberman) [SIG API Machinery and Apps]

• Remove deprecated --enable-taint-manager and --pod-eviction-timeout CLI (kubernetes/kubernetes#115840, @atosatto)

• Removed support for the v1alpha1 kubeletplugin API of DynamicResourceManagement. All plugins must be updated to v1alpha2 in order to function properly. (kubernetes/kubernetes#116558, @klueska)

• The API server now re-uses data encryption keys while the kms v2 plugin key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. (kubernetes/kubernetes#116155, @enj)

• The PodDisruptionBudget spec.unhealthyPodEvictionPolicy field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to AlwaysAllow to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. (kubernetes/kubernetes#115363, @ravisantoshgudimetla) [SIG Apps, Auth and Node]

• The DownwardAPIHugePages kubelet feature graduated to stable / GA. (kubernetes/kubernetes#115721, @saschagrunert) [SIG Apps and Node]

• The following feature gates for volume expansion GA features have now been removed and must no longer be referenced in --feature-gates flags: ExpandCSIVolumes, ExpandInUsePersistentVolumes, ExpandPersistentVolumes (kubernetes/kubernetes#113942, @mengjiao-liu)

• The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from set to map, resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. (kubernetes/kubernetes#114585, @JoelSpeed)

• Updated API reference for Requests, specifying they must not exceed limits (kubernetes/kubernetes#115434, @ehashman)

• Updated KMSv2 to beta (kubernetes/kubernetes#115123, @aramase)

• Updated: Redefine AppProtocol field description and add new standard values (kubernetes/kubernetes#115433, @LiorLieberman) [SIG API Machinery, Apps and Network]

• /metrics/slis is now available for control plane components allowing you to scrape health check metrics. (kubernetes/kubernetes#114997, @Richabanker)

• APIServerTracing feature gate is now enabled by default. Tracing in the API
  Server is still disabled by default, and requires a config file to enable. (kubernetes/kubernetes#116144, @dashpole)

• NodeResourceFit and NodeResourcesBalancedAllocation implement the PreScore
  extension point for a more performant calculation. (kubernetes/kubernetes#115655, @tangwz)

• PodSchedulingReadiness is graduated to beta. (kubernetes/kubernetes#115815, @Huang-Wei)

• PodSpec.Container.Resources became mutable for CPU and memory resource types.

  • PodSpec.Container.ResizePolicy (new object) gives users control over how their containers are resized.
  • PodStatus.Resize status describes the state of a requested Pod resize.
  • PodStatus.ResourcesAllocated describes node resources allocated to Pod.
  • PodStatus.Resources describes node resources applied to running containers by CRI.
  • UpdateContainerResources CRI API now supports both Linux and Windows. (kubernetes/kubernetes#102884, @vinaykul)

• SELinuxMountReadWriteOncePod graduated to Beta. (kubernetes/kubernetes#116425, @jsafrane)

• StatefulSetAutoDeletePVC feature gate promoted to beta. (kubernetes/kubernetes#116501, @mattcary)

• StatefulSet names must be DNS labels, rather than subdomains. Any StatefulSet
  which took advantage of subdomain validation (by having dots in the name) can't
  possibly have worked, because we eventually set pod.spec.hostname from the StatefulSetName,
  and that is validated as a DNS label. (kubernetes/kubernetes#114172, @thockin)

• ValidatingAdmissionPolicy now provides a status field that contains results of type checking the validation expression.
  The type checking is fully informational, and the behavior of the policy is unchanged. (kubernetes/kubernetes#115668, @jiahuif)

• cacheSize field in EncryptionConfiguration is not supported for KMSv2 provider (kubernetes/kubernetes#113121, @aramase)

• k8s.io/component-base/logs now also supports adding command line flags to a flag.FlagSet. (kubernetes/kubernetes#114731, @pohly)

• kubelet: migrated --container-runtime-endpoint and --image-service-endpoint
  to kubelet config (kubernetes/kubernetes#112136, @pacoxu)

• resource.k8s.io/v1alpha1 was replaced with resource.k8s.io/v1alpha2. Before
  upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate,
  ResourceClass, PodScheduling) must be deleted. The changes are internal, so
  YAML files which create pods and resource claims don't need changes except for
  the newer apiVersion. (kubernetes/kubernetes#116299, @pohly)

• volumes: resource.claims is now cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. (kubernetes/kubernetes#115928, @pohly)

• Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
  A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (kubernetes/kubernetes#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]

• Remove kubernetes.io/grpc standard appProtocol (kubernetes/kubernetes#116866, @LiorLieberman) [SIG API Machinery and Apps]

• API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. (kubernetes/kubernetes#116556, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]

• APIServerTracing feature gate is now enabled by default. Tracing in the API Server is still disabled by default, and requires a config file to enable. (kubernetes/kubernetes#116144, @dashpole) [SIG API Machinery and Testing]

• Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
  restrictions that already apply to CustomResourceDefinition.
  If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
  admission check that was being performed is aborted; the failurePolicy for the ValidatingAdmissionPolicy
  determines the outcome. (kubernetes/kubernetes#115747, @cici37) [SIG API Machinery]

• Added messageExpression to ValidatingAdmissionPolicy, to set custom failure message via CEL expression. (kubernetes/kubernetes#116397, @jiahuif) [SIG API Machinery]

• Added a new IPAddress object kind

  • Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 (kubernetes/kubernetes#115075, @aojea) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]

• Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
  A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (kubernetes/kubernetes#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]

• Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a authorizer
  variable with expressions. The new variable provides a builder that allows expressions such authorizer.group('').resource('pods').check('create').allowed(). (kubernetes/kubernetes#116054, @jpbetz) [SIG API Machinery and Testing]

• Added matchConditions field to ValidatingAdmissionPolicy, enabled support for CEL based custom match criteria. (kubernetes/kubernetes#116350, @maxsmythe) [SIG API Machinery and Testing]

• Added messageExpression field to ValidationRule. (#115969, @DangerOnTheRanger) (kubernetes/kubernetes#115969, @DangerOnTheRanger) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node and Testing]

• Added the MatchConditions field to ValidatingWebhookConfiguration and MutatingWebhookConfiguration for the v1beta and v1 apis.

  The AdmissionWebhookMatchConditions featuregate is now in Alpha (kubernetes/kubernetes#116261, @ivelichkovich) [SIG API Machinery and Testing]

• Added validation to ensure that if service.kubernetes.io/topology-aware-hints and service.kubernetes.io/topology-mode annotations are both set, they are set to the same value.

  • Added deprecation warning if service.kubernetes.io/topology-aware-hints annotation is used. (kubernetes/kubernetes#116612, @robscott) [SIG Apps, Network and Testing]

• Adds auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
  Adds validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. (kubernetes/kubernetes#115973, @jpbetz) [SIG API Machinery and Testing]

• Adds feature gate NodeLogQuery which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. (kubernetes/kubernetes#96120, @LorbusChris) [SIG API Machinery, Apps, CLI, Node, Testing and Windows]

• Api: validation of a PodSpec now rejects invalid ResourceClaim and ResourceClaimTemplate names. For a pod, the name generated for the ResourceClaim when using a template also must be valid. (kubernetes/kubernetes#116576, @pohly) [SIG Apps]

• Bump default API QPS limits for Kubelet. (kubernetes/kubernetes#116121, @wojtek-t) [SIG API Machinery and Node]

• Enable the "StatefulSetStartOrdinal" feature gate in beta (kubernetes/kubernetes#115260, @pwschuurman) [SIG API Machinery and Apps]

• Extended the kubelet's PodResources API to include resources allocated in ResourceClaims via DynamicResourceAllocation. Additionally, added a new Get() method to query a specific pod for its resources. (kubernetes/kubernetes#115847, @moshe010) [SIG Node]

• Forbid to set matchLabelKeys when labelSelector isn’t set in topologySpreadConstraints (kubernetes/kubernetes#116535, @denkensk) [SIG API Machinery, Apps and Scheduling]

• GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) (kubernetes/kubernetes#115966, @aojea) [SIG Apps and Cloud Provider]

• GRPC probes are now a GA feature. GRPCContainerProbe feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. (kubernetes/kubernetes#116233, @SergeyKanzhelev) [SIG API Machinery, Apps and Node]

• Graduate Kubelet Topology Manager to GA. (kubernetes/kubernetes#116093, @swatisehgal) [SIG API Machinery, Node and Testing]

• Graduate KubeletTracing to beta, which means that the feature gate is now enabled by default. (kubernetes/kubernetes#115750, @saschagrunert) [SIG Instrumentation and Node]

• Graduate the container resource metrics feature on HPA to beta. (kubernetes/kubernetes#116046, @sanposhiho) [SIG Autoscaling]

• Introduced a breaking change to the resource.k8s.io API in its AllocationResult struct. This change allows a kubelet plugin for the DynamicResourceAllocation feature to service allocations from multiple resource driver controllers. (kubernetes/kubernetes#116332, @klueska) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]

• Introduces new alpha functionality to the reflector, allowing user to enable API streaming.

  To activate this feature, users can set the ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environmental variable.
  It is important to note that the server must support streaming for this feature to function properly.
  If streaming is not supported by the server, the reflector will revert to the previous method
  of obtaining data through LIST/WATCH semantics. (kubernetes/kubernetes#110772, @p0lyn0mial) [SIG API Machinery]

• Kubelet: change MemoryThrottlingFactor default value to 0.9 and formulas to calculate memory.high (kubernetes/kubernetes#115371, @pacoxu) [SIG API Machinery, Apps and Node]

• Migrated the DaemonSet controller (within `kube-controller-manager) to use contextual logging (kubernetes/kubernetes#113622, @249043822) [SIG API Machinery, Apps, Instrumentation and Testing]

• New service.kubernetes.io/topology-mode annotation has been introduced as a replacement for the service.kubernetes.io/topology-aware-hints annotation.

  • service.kubernetes.io/topology-aware-hints annotation has been deprecated.
  • kube-proxy now accepts any value that is not "disabled" for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. (kubernetes/kubernetes#116522, @robscott) [SIG Apps, Network and Testing]

• NodeResourceFit and NodeResourcesBalancedAllocation implement the PreScore extension point for a more performant calculation. (kubernetes/kubernetes#115655, @tangwz) [SIG Scheduling]

• Pods owned by a Job will now use the labels batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid.
  The legacy labels job-name and controller-uid are still added for compatibility. (kubernetes/kubernetes#114930, @kannon92) [SIG Apps]

• Promote CronJobTimeZone feature to GA (kubernetes/kubernetes#115904, @soltysh) [SIG API Machinery and Apps]

• Promoted SelfSubjectReview to Beta (kubernetes/kubernetes#116274, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]

• Relax API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). (kubernetes/kubernetes#116161, @danielvegamyhre) [SIG Apps, Scheduling and Testing]

• Remove deprecated --enable-taint-manager and --pod-eviction-timeout CLI flags (kubernetes/kubernetes#115840, @atosatto) [SIG API Machinery, Apps, Node and Testing]

• Resource.k8s.io/v1alpha1 was replaced with resource.k8s.io/v1alpha2. Before upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate, ResourceClass, PodScheduling) must be deleted. The changes will be internal, so YAML files which create pods and resource claims don't need changes except for the newer apiVersion. (kubernetes/kubernetes#116299, @pohly) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]

• SELinuxMountReadWriteOncePod graduated to Beta. (kubernetes/kubernetes#116425, @jsafrane) [SIG Storage and Testing]

• StatefulSetAutoDeletePVC feature gate promoted to beta. (kubernetes/kubernetes#116501, @mattcary) [SIG Apps, Auth and Testing]

• The API server now re-uses data encryption keys while the kms v2 plugin's key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. (kubernetes/kubernetes#116155, @enj) [SIG API Machinery, Auth and Testing]

• The API server's encryption at rest configuration now allows the use of wildcards in the list of resources. For example, '.' can be used to encrypt all resources, including all current and future custom resources. (kubernetes/kubernetes#115149, @nilekhc) [SIG API Machinery, Auth and Testing]

• Update KMSv2 to beta (kubernetes/kubernetes#115123, @aramase) [SIG API Machinery, Auth and Testing]

• Updated: Redefine AppProtocol field description and add new standard values (kubernetes/kubernetes#115433, @LiorLieberman) [SIG API Machinery, Apps and Network]

• ValidatingAdmissionPolicy now provides a status field that contains results of type checking the validation expression.
  The type checking is fully informational, and the behavior of the policy is unchanged. (kubernetes/kubernetes#115668, @jiahuif) [SIG API Machinery, Auth, Cloud Provider and Testing]

• We have removed support for the v1alpha1 kubeletplugin API of DynamicResourceManagement. All plugins must update to v1alpha2 in order to function properly going forward. (kubernetes/kubernetes#116558, @klueska) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]

• Graduated seccomp profile defaulting to GA.

  Set the kubelet --seccomp-default flag or seccompDefault kubelet configuration field to true to make pods on that node default to using the RuntimeDefault seccomp profile.

  Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.

  Guidance for identifying and mitigating those issues is outlined in the Kubernetes seccomp tutorial. (kubernetes/kubernetes#115719, @saschagrunert) [SIG API Machinery, Node, Storage and Testing]

• Implements API for streaming for the watch-cache

  When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (kubernetes/kubernetes#110960, @p0lyn0mial) [SIG API Machinery]

• Introduce API for streaming.

  Add SendInitialEvents field to the ListOptions. When the new option is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (kubernetes/kubernetes#115402, @p0lyn0mial) [SIG API Machinery]

• Kubelet: a "maxParallelImagePulls" field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. (kubernetes/kubernetes#115220, @ruiwen-zhao) [SIG API Machinery, Node and Scalability]

• PodSchedulingReadiness is graduated to beta. (kubernetes/kubernetes#115815, @Huang-Wei) [SIG API Machinery, Apps, Scheduling and Testing]

• In-place resize feature for Kubernetes Pods

  • Changed the Pod API so that the resources defined for containers are mutable for cpu and memory resource types.
  • Added resizePolicy for containers in a pod to allow users control over how their containers are resized.
  • Added allocatedResources field to container status in pod status that describes the node resources allocated to a pod.
  • Added resources field to container status that reports actual resources applied to running containers.
  • Added resize field to pod status that describes the state of a requested pod resize.
    For details, see KEPs below. (kubernetes/kubernetes#102884, @vinaykul) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling and Testing]

• The PodDisruptionBudget spec.unhealthyPodEvictionPolicy field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to AlwaysAllow to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. (kubernetes/kubernetes#115363, @ravisantoshgudimetla) [SIG Apps, Auth and Node]

• The DownwardAPIHugePages kubelet feature graduated to stable / GA. (kubernetes/kubernetes#115721, @saschagrunert) [SIG Apps and Node]

• Volumes: resource.claims gets cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. (kubernetes/kubernetes#115928, @pohly) [SIG API Machinery, Apps and Storage]

• A fix in the resource.k8s.io/v1alpha1/ResourceClaim API avoids harmless (?) ".status.reservedFor: element 0: associative list without keys has an element that's a map type" errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. (kubernetes/kubernetes#115354, @pohly) [SIG API Machinery]

• CacheSize field in EncryptionConfiguration is not supported for KMSv2 provider (kubernetes/kubernetes#113121, @aramase) [SIG API Machinery, Auth and Testing]

• K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. (kubernetes/kubernetes#115514, @pohly) [SIG API Machinery]

• K8s.io/component-base/logs now also supports adding command line flags to a flag.FlagSet. (kubernetes/kubernetes#114731, @pohly) [SIG Architecture]

• Update API reference for Requests, specifying they must not exceed limits (kubernetes/kubernetes#115434, @ehashman) [SIG Architecture, Docs and Node]

• /metrics/slis is made available for control plane components allowing you to scrape health check metrics. (kubernetes/kubernetes#114997, @Richabanker) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]

• A terminating pod on a node that is not caused by preemption won't prevent kube-scheduler from preempting pods on that node

  • Rename 'PreemptionByKubeScheduler' to 'PreemptionByScheduler' (kubernetes/kubernetes#114623, @Huang-Wei) [SIG Scheduling]

• Added new option to the InterPodAffinity scheduler plugin to ignore existing pods` preferred inter-pod affinities if the incoming pod has no preferred inter-pod affinities. This option can be used as an optimization for higher scheduling throughput (at the cost of an occasional pod being scheduled non-optimally/violating existing pods' preferred inter-pod affinities). To enable this scheduler option, set the InterPodAffinity scheduler plugin arg "ignorePreferredTermsOfExistingPods: true". (kubernetes/kubernetes#114393, @danielvegamyhre) [SIG API Machinery and Scheduling]

• Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. (kubernetes/kubernetes#114412, @thockin) [SIG API Machinery and Apps]

• K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message (kubernetes/kubernetes#114680, @pohly) [SIG Instrumentation]

• Kube-proxy, kube-scheduler and kubelet have HTTP APIs for changing the logging verbosity at runtime. This now also works for JSON output. (kubernetes/kubernetes#114609, @pohly) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation and Testing]

• Kubeadm: explicitly set priority for static pods with priorityClassName: system-node-critical (kubernetes/kubernetes#114338, @champtar) [SIG Cluster Lifecycle]

• Kubelet: migrate "--container-runtime-endpoint" and "--image-service-endpoint" to kubelet config (kubernetes/kubernetes#112136, @pacoxu) [SIG API Machinery, Node and Scalability]

• Kubernetes components that perform leader election now only support using Leases for this. (kubernetes/kubernetes#114055, @aimuz) [SIG API Machinery, Cloud Provider and Scheduling]

• StatefulSet names must be DNS labels, rather than subdomains. Any StatefulSet which took advantage of subdomain validation (by having dots in the name) can't possibly have worked, because we eventually set pod.spec.hostname from the StatefulSetName, and that is validated as a DNS label. (kubernetes/kubernetes#114172, @thockin) [SIG Apps]

• The following feature gates for volume expansion GA features have been removed and must no longer be referenced in --feature-gates flags: ExpandCSIVolumes, ExpandInUsePersistentVolumes, ExpandPersistentVolumes (kubernetes/kubernetes#113942, @mengjiao-liu) [SIG API Machinery, Apps and Testing]

• The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from "set" to "map", resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. (kubernetes/kubernetes#114585, @JoelSpeed) [SIG API Machinery]

v26.1.0

Kubernetes API Version: v1.26.1

Bug or Regression Copy Section

• The timeout unit of the WSClient update method is now always seconds for both poll and select functions. (#1976, @t-yrka)

Feature Copy Section

• Adds support for loading CA certificates from a file using the idp-certificate-authority key for the oidc plugin. (#1916, @vgupta3)

v26.1.0b1

Kubernetes API Version: v1.26.1

Bug or Regression Copy Section

• The timeout unit of the WSClient update method is now always seconds for both poll and select functions. (#1976, @t-yrka)

Feature Copy Section

• Adds support for loading CA certificates from a file using the idp-certificate-authority key for the oidc plugin. (#1916, @vgupta3)

v26.1.0a1

Kubernetes API Version: v1.26.1

API Change Copy Section

• The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from "set" to "map", resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. (kubernetes/kubernetes#114617, @JoelSpeed) [SIG API Machinery]

• 'A new preEnqueue extension point was added to scheduler's component config
  v1beta2/v1beta3/v1.'
  (kubernetes/kubernetes#113275, @Huang-Wei)

• 'Added a ResourceClaim API (in the resource.k8s.io/v1alpha1 API group and
  behind the DynamicResourceAllocation feature gate).
  The new API is now more flexible than the existing Device Plugins feature of Kubernetes because it
  allows Pods to request (claim) special kinds of resources, which can be available at node level, cluster
  level, or following any other model you implement.' (kubernetes/kubernetes#111023, @pohly)

• 'Container preStop and postStart lifecycle handlers using httpGet now
  honor the specified scheme and headers fields. This enables setting custom
  headers and changing the scheme to HTTPS, consistent with container
  startup/readiness/liveness probe capabilities. Lifecycle handlers configured
  with scheme: HTTPS that encounter errors indicating the endpoint is actually
  using HTTP fall back to making the request over HTTP for compatibility with
  previous releases. When this happens, a LifecycleHTTPFallback event is recorded
  in the namespace of the pod and a kubelet_lifecycle_handler_http_fallbacks_total
  metric in the kubelet is incremented. Cluster administrators can opt out of the
  expanded lifecycle handler capabilities by setting
  --feature-gates=ConsistentHTTPGetHandlers=false in kubelet.'
  (kubernetes/kubernetes#86139, @jasimmons)

• 'Graduated JobTrackingWithFinalizers to stable.
  Jobs created before the feature was enabled are still tracked without finalizers.
  Jobs tracked with finalizers have the annotation batch.kubernetes.io/job-tracking.
  If the annotation is present and the user attempts to remove it, the control plane adds it back.
  The annotation batch.kubernetes.io/job-tracking is now deprecated.
  The control plane will ignore it and stop adding it for new Jobs in v1.27.' (kubernetes/kubernetes#113510, @alculquicondor)

• 'Kubelet added the following Pod failure conditions:

  • DisruptionTarget (graceful node shutdown, node pressure eviction)' (kubernetes/kubernetes#112360, @mimowo)

• 'Priority and Fairness has introduced a new feature called borrowing that allows an API priority level
  to borrow a number of seats from other priority level(s). As a cluster operator, you can enable borrowing
  for a certain priority level configuration object via the two newly introduced fields lendablePercent, and
  borrowingLimitPercent located under the .spec.limited field of the designated priority level.
  This change added the following metrics:

  • apiserver_flowcontrol_nominal_limit_seats: Nominal number of execution seats configured for each priority level
  • apiserver_flowcontrol_lower_limit_seats: Configured lower bound on number of execution seats available to each priority level
  • apiserver_flowcontrol_upper_limit_seats: Configured upper bound on number of execution seats available to each priority level
  • apiserver_flowcontrol_demand_seats: Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)
  • apiserver_flowcontrol_demand_seats_high_watermark: High watermark, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_average: Time-weighted average, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_stdev: Time-weighted standard deviation, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_smoothed: Smoothed seat demands
  • apiserver_flowcontrol_target_seats: Seat allocation targets
  • apiserver_flowcontrol_seat_fair_frac: Fair fraction of server's concurrency to allocate to each priority level that can use it
  • apiserver_flowcontrol_current_limit_seats: current derived number of execution seats available to each priority level
    The possibility of borrowing means that the old metric apiserver_flowcontrol_request_concurrency_limit can no longer mean both the configured concurrency limit and the enforced concurrency limit. Henceforth it means the configured concurrency limit.' (kubernetes/kubernetes#113485, @MikeSpreitzer)

• 'NodeInclusionPolicy in podTopologySpread plugin is now enabled by default.'
  (kubernetes/kubernetes#113500, @kerthcet)

• 'PodDisruptionBudget now adds an alpha spec.unhealthyPodEvictionPolicy field.
  When the PDBUnhealthyPodEvictionPolicy feature-gate is enabled in kube-apiserver,
  setting this field to "AlwaysAllow" allows pods to be evicted if they do not
  have a ready condition, regardless of whether the PodDisruptionBudget is currently
  healthy.'
  (kubernetes/kubernetes#113375, @atiratree)

• 'metav1.LabelSelectors specified in API objects are now validated to ensure
  they do not contain invalid label values that will error at time of use. Existing
  invalid objects can be updated, but new objects are required to contain valid
  label selectors.'
  (kubernetes/kubernetes#113699, @liggitt)

• Add percentageOfNodesToScore as a scheduler profile level parameter to API version v1. When a profile percentageOfNodesToScore is set, it will override global percentageOfNodesToScore. (kubernetes/kubernetes#112521, @yuanchen8911)

• Add auth API to get self subject attributes (new selfsubjectreviews API is added).
  The corresponding command for kubctl is provided - kubectl auth whoami. (kubernetes/kubernetes#111333, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]

• Added kubernetes_feature_enabled metric series to track whether each active feature gate is enabled. (kubernetes/kubernetes#112690, @logicalhan)

• Added a --topology-manager-policy-options flag to the kubelet to support fine tuning the topology manager policies. The first policy option, prefer-closest-numa-nodes, allows these policies to favor sets of NUMA nodes with shorter distance between nodes when making admission decisions. (kubernetes/kubernetes#112914, @PiotrProkop)

• Added a feature that allows a StatefulSet to start numbering replicas from an arbitrary non-negative ordinal, using the .spec.ordinals.start field. (kubernetes/kubernetes#112744, @pwschuurman)

• Added a kube-proxy flag (--iptables-localhost-nodeports, default true) to allow disabling NodePort services on loopback addresses. Note: this only applies to iptables mode and ipv4. (kubernetes/kubernetes#108250, @cyclinder)

• Added a new namespace alpha field to DataSourceRef field in PersistentVolumeClaim API. (kubernetes/kubernetes#113186, @ttakahashi21)

• Aggregated discovery will be alpha and can be toggled with the AggregatedDiscoveryEndpoint feature flag. (kubernetes/kubernetes#113171, @Jefftree)

• Clarified the CFS quota as 100ms in the code comments and set the minimum cpuCFSQuotaPeriod to 1ms to match Linux kernel expectations. (kubernetes/kubernetes#112123, @paskal)

• Component-base: make the validation logic about LeaderElectionConfiguration consistent between component-base and client-go (kubernetes/kubernetes#111758, @SataQiu) [SIG API Machinery and Scheduling]

• Deprecated the apiserver_request_slo_duration_seconds metric for v1.27 in favor of apiserver_request_sli_duration_seconds for naming consistency purposes with other SLI-specific metrics and to avoid any confusion between SLOs and SLIs. (kubernetes/kubernetes#112679, @dgrisonnet)

• Enable the "Retriable and non-retriable pod failures for jobs" feature into beta. (kubernetes/kubernetes#113360, @mimowo)

• Enabled kube-controller-manager to support '--concurrent-horizontal-pod-autoscaler-syncs' flag to set the number of horizontal pod autoscaler controller workers. (kubernetes/kubernetes#108501, @zroubalik)

• Fixed spurious field is immutable errors validating updates to Event API objects via the events.k8s.io/v1 API. (kubernetes/kubernetes#112183, @liggitt)

• Graduated ServiceInternalTrafficPolicy feature to GA. (kubernetes/kubernetes#113496, @avoltz)

• In 'kube-proxy`: The "userspace" proxy mode (deprecated for over a year) is no
  longer supported on either Linux or Windows. Users should use "iptables" or "ipvs"
  on Linux, or "kernelspace" on Windows.
  (kubernetes/kubernetes#112133, @knabben)

• Introduce v1beta3 for Priority and Fairness with the following changes to the API spec:

  • rename 'assuredConcurrencyShares' (located under `spec.limited') to 'nominalConcurrencyShares'.
  • apply strategic merge patch annotations to 'Conditions' of flowschemas and prioritylevelconfigurations. (kubernetes/kubernetes#112306, @tkashem)

• Introduced v1alpha1 API for validating admission policies, enabling extensible admission control via CEL expressions (KEP 3488: CEL for Admission Control). To use, enable the ValidatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config. (kubernetes/kubernetes#113314, @cici37)

• KMS: added validation for duplicate kms config name when auto reload is enabled. If you enabled automatic reload of encryption configuration with API server flag --encryption-provider-config-automatic-reload, ensure all the KMS provider names (v1 and v2) in the encryption configuration are unique. (kubernetes/kubernetes#113697, @aramase)

• Kubelet external Credential Provider feature is moved to GA. Credential Provider Plugin and Credential Provider Config APIs updated from v1beta1 to v1 with no API changes. (kubernetes/kubernetes#111616, @ndixita)

• Legacy klog flags are no longer available. Only -v and -vmodule are still supported. (kubernetes/kubernetes#112120, @pohly) [SIG Architecture, CLI, Instrumentation, Node and Testing]

• Moved MixedProtocolLBService from beta to GA. (kubernetes/kubernetes#112895, @janosi)

• New Pod API field .spec.schedulingGates is introduced to enable users to control when to mark a Pod as scheduling ready. (kubernetes/kubernetes#113274, @Huang-Wei)

• Protobuf serialization of metav1.MicroTime timestamps (used in Lease and Event API objects) has been corrected to truncate to microsecond precision, to match the documented behavior and JSON/YAML serialization. Any existing persisted data is truncated to microsecond when read from etcd. (kubernetes/kubernetes#111936, @haoruan)

• Removed feature gates ServiceLoadBalancerClass and ServiceLBNodePortControl. These feature gates were enabled (and locked) since v1.24. (kubernetes/kubernetes#112577, @andrewsykim)

• Reverted regression that prevented client-go latency metrics to be reported with a template URL to avoid label cardinality. (kubernetes/kubernetes#111752, @aanm)

• The EndpointSliceTerminatingCondition feature gate was graduated to GA. The gate is now locked and will be removed in v1.28. (kubernetes/kubernetes#113351, @andrewsykim)

• DynamicKubeletConfig feature gate has been removed from the API server.
  Dynamic kubelet reconfiguration now can't be used even when older nodes are still
  attempting to rely on it. This is aligned with the Kubernetes version skew policy.
  (kubernetes/kubernetes#112643, @SergeyKanzhelev)

• kubectl wait command with jsonpath flag will wait for target path until timeout.
  (kubernetes/kubernetes#109525, @jonyhy96)

• Add a ResourceClaim API (in the resource.k8s.io/v1alpha1 API group and
  behind the DynamicResourceAllocation feature gate).
  The new API is more flexible than the existing Device Plugins feature of Kubernetes because it
  allows Pods to request (claim) special kinds of resources, which can be available at node level, cluster
  level, or following any other model you implement. (kubernetes/kubernetes#111023, @pohly) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release, Scheduling, Storage and Testing]

• PodDisruptionBudget adds an alpha spec.unhealthyPodEvictionPolicy field. When the PDBUnhealthyPodEvictionPolicy feature-gate is enabled in kube-apiserver, setting this field to "AlwaysAllow" allows pods to be evicted if they do not have a ready condition, regardless of whether the PodDisruptionBudget is currently healthy. (kubernetes/kubernetes#113375, @atiratree) [SIG API Machinery, Apps, Auth and Testing]

• A new preEnqueue extension point is added to scheduler's component config v1beta2/v1beta3/v1. (kubernetes/kubernetes#113275, @Huang-Wei) [SIG API Machinery, Apps, Instrumentation, Scheduling and Testing]

• Add a new namespace alpha field to dataSourceRef field in PersistentVolumeClaim API. (kubernetes/kubernetes#113186, @ttakahashi21) [SIG API Machinery, Apps, Storage and Testing]

• Add a kube-proxy flag (--iptables-localhost-nodeports, default true) to allow disabling NodePort services on loopback addresses. Note: this only applies to iptables mode and ipv4. (kubernetes/kubernetes#108250, @cyclinder) [SIG API Machinery, Cloud Provider, Network, Node, Scalability, Storage and Testing]

• Added a --topology-manager-policy-options flag to the kubelet to support fine tuning the topology manager policies. The first policy option, prefer-closest-numa-nodes, allows these policies to favor sets of NUMA nodes with shorter distance between nodes when making admission decisions. (kubernetes/kubernetes#112914, @PiotrProkop) [SIG API Machinery and Node]

• Added a feature that allows a StatefulSet to start numbering replicas from an arbitrary non-negative ordinal, using the .spec.ordinals.start field. (kubernetes/kubernetes#112744, @pwschuurman) [SIG API Machinery and Apps]

• Deprecate the apiserver_request_slo_duration_seconds metric for v1.27 in favor of apiserver_request_sli_duration_seconds for naming consistency purposes with other SLI-specific metrics and to avoid any confusion between SLOs and SLIs. (kubernetes/kubernetes#112679, @dgrisonnet) [SIG API Machinery and Instrumentation]

• Enable the "Retriable and non-retriable pod failures for jobs" feature into beta (kubernetes/kubernetes#113360, @mimowo) [SIG Apps, Auth, Node, Scheduling and Testing]

• Graduate JobTrackingWithFinalizers to stable.
  Jobs created before the feature was enabled are still tracked without finalizers.
  Users can choose to migrate jobs to tracking with finalizers by adding the annotation batch.kubernetes.io/job-tracking.
  If the annotation was already present and the user attempts to remove it, the control plane adds the annotation back. (kubernetes/kubernetes#113510, @alculquicondor) [SIG API Machinery, Apps and Testing]

• Graduate ServiceInternalTrafficPolicy feature to GA (kubernetes/kubernetes#113496, @avoltz) [SIG Apps and Network]

• If you enabled automatic reload of encryption configuration with API server flag --encryption-provider-config-automatic-reload, ensure all the KMS provider names (v1 and v2) in the encryption configuration are unique. (kubernetes/kubernetes#113697, @aramase) [SIG API Machinery and Auth]

• Introduce v1alpha1 API for validating admission policies, enabling extensible admission control via CEL expressions (KEP 3488: CEL for Admission Control). To use, enable the ValidatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config. (kubernetes/kubernetes#113314, @cici37) [SIG API Machinery, Auth, Cloud Provider and Testing]

• Kubelet adds the following pod failure conditions:

  • DisruptionTarget (graceful node shutdown, node pressure eviction) (kubernetes/kubernetes#112360, @mimowo) [SIG Apps, Node and Testing]

• Metav1.LabelSelectors specified in API objects are now validated to ensure they do not contain invalid label values that will error at time of use. Existing invalid objects can be updated, but new objects are required to contain valid label selectors. (kubernetes/kubernetes#113699, @liggitt) [SIG API Machinery, Apps, Auth, Network and Storage]

• Moving MixedProtocolLBService from beta to GA (kubernetes/kubernetes#112895, @janosi) [SIG Apps, Network and Testing]

• New Pod API field .spec.schedulingGates is introduced to enable users to control when to mark a Pod as scheduling ready. (kubernetes/kubernetes#113274, @Huang-Wei) [SIG Apps, Scheduling and Testing]

• NodeInclusionPolicy in podTopologySpread plugin is enabled by default. (kubernetes/kubernetes#113500, @kerthcet) [SIG API Machinery, Apps, Scheduling and Testing]

• Priority and Fairness has introduced a new feature called borrowing that allows an API priority level
  to borrow a number of seats from other priority level(s). As a cluster operator, you can enable borrowing
  for a certain priority level configuration object via the two newly introduced fields lendablePercent, and
  borrowingLimitPercent located under the .spec.limited field of the designated priority level.
  This PR adds the following metrics.

  • apiserver_flowcontrol_nominal_limit_seats: Nominal number of execution seats configured for each priority level
  • apiserver_flowcontrol_lower_limit_seats: Configured lower bound on number of execution seats available to each priority level
  • apiserver_flowcontrol_upper_limit_seats: Configured upper bound on number of execution seats available to each priority level
  • apiserver_flowcontrol_demand_seats: Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)
  • apiserver_flowcontrol_demand_seats_high_watermark: High watermark, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_average: Time-weighted average, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_stdev: Time-weighted standard deviation, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_smoothed: Smoothed seat demands
  • apiserver_flowcontrol_target_seats: Seat allocation targets
  • apiserver_flowcontrol_seat_fair_frac: Fair fraction of server's concurrency to allocate to each priority level that can use it
  • apiserver_flowcontrol_current_limit_seats: current derived number of execution seats available to each priority level

  The possibility of borrowing means that the old metric apiserver_flowcontrol_request_concurrency_limit can no longer mean both the configured concurrency limit and the enforced concurrency limit. Henceforth it means the configured concurrency limit. (kubernetes/kubernetes#113485, @MikeSpreitzer) [SIG API Machinery and Testing]

• The EndpointSliceTerminatingCondition feature gate has graduated to GA. The gate is now locked and will be removed in v1.28. (kubernetes/kubernetes#113351, @andrewsykim) [SIG API Machinery, Apps, Network and Testing]

• Yes, aggregated discovery will be alpha and can be toggled with the AggregatedDiscoveryEndpoint feature flag (kubernetes/kubernetes#113171, @Jefftree) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Network, Node, Release, Scalability, Scheduling, Storage and Testing]

• Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

  (kubernetes/kubernetes#86139, @jasimmons) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Contributor Experience, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]

• Add percentageOfNodesToScore as a scheduler profile level parameter to API version v1. If a profile percentageOfNodesToScore is set, it will override global percentageOfNodesToScore. (kubernetes/kubernetes#112521, @yuanchen8911) [SIG API Machinery, Scheduling and Testing]

• Kube-controller-manager supports '--concurrent-horizontal-pod-autoscaler-syncs' flag to set the number of horizontal pod autoscaler controller workers. (kubernetes/kubernetes#108501, @zroubalik) [SIG API Machinery, Apps and Autoscaling]

• Kube-proxy: The "userspace" proxy mode (deprecated for over a year) is no longer supported on either Linux or Windows. Users should use "iptables" or "ipvs" on Linux, or "kernelspace" on Windows. (kubernetes/kubernetes#112133, @knabben) [SIG API Machinery, Network, Scalability, Testing and Windows]

• Kubectl wait command with jsonpath flag will wait for target path appear until timeout. (kubernetes/kubernetes#109525, @jonyhy96) [SIG CLI and Testing]

• Kubelet external Credential Provider feature is moved to GA. Credential Provider Plugin and Credential Provider Config APIs updated from v1beta1 to v1 with no API changes. (kubernetes/kubernetes#111616, @ndixita) [SIG API Machinery, Node, Scheduling and Testing]

• The DynamicKubeletConfig feature gate has been removed from the API server. Dynamic kubelet reconfiguration now cannot be used even when older nodes are still attempting to rely on it. This is aligned with the Kubernetes version skew policy. (kubernetes/kubernetes#112643, @SergeyKanzhelev) [SIG API Machinery, Apps, Auth, Node and Testing]

• Add kubernetes_feature_enabled metric series to track whether each active feature gate is enabled. (kubernetes/kubernetes#112690, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle, Instrumentation, Network, Node and Scheduling]

• Introduce v1beta3 for Priority and Fairness with the following changes to the API spec:

  • rename 'assuredConcurrencyShares' (located under spec.limited') to 'nominalConcurrencyShares'
  • apply strategic merge patch annotations to 'Conditions' of flowschemas and prioritylevelconfigurations (kubernetes/kubernetes#112306, @tkashem) [SIG API Machinery and Testing]

• Legacy klog flags are no longer available. Only -v and -vmodule are still supported. (kubernetes/kubernetes#112120, @pohly) [SIG Architecture, CLI, Instrumentation, Node and Testing]

• The feature gates ServiceLoadBalancerClass and ServiceLBNodePortControl have been removed. These feature gates were enabled (and locked) since v1.24. (kubernetes/kubernetes#112577, @andrewsykim) [SIG Apps]

• Add auth API to get self subject attributes (new selfsubjectreviews API is added).
  The corresponding command for kubctl is provided - kubectl auth whoami. (kubernetes/kubernetes#111333, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]

• Clarified the CFS quota as 100ms in the code comments and set the minimum cpuCFSQuotaPeriod to 1ms to match Linux kernel expectations. (kubernetes/kubernetes#112123, @paskal) [SIG API Machinery and Node]

• Component-base: make the validation logic about LeaderElectionConfiguration consistent between component-base and client-go (kubernetes/kubernetes#111758, @SataQiu) [SIG API Machinery and Scheduling]

• Fixes spurious field is immutable errors validating updates to Event API objects via the events.k8s.io/v1 API (kubernetes/kubernetes#112183, @liggitt) [SIG Apps]

• Protobuf serialization of metav1.MicroTime timestamps (used in Lease and Event API objects) has been corrected to truncate to microsecond precision, to match the documented behavior and JSON/YAML serialization. Any existing persisted data is truncated to microsecond when read from etcd. (kubernetes/kubernetes#111936, @haoruan) [SIG API Machinery]

• Revert regression that prevented client-go latency metrics to be reported with a template URL to avoid label cardinality. (kubernetes/kubernetes#111752, @aanm) [SIG API Machinery]

• [kubelet] Change default cpuCFSQuotaPeriod value with enabled cpuCFSQuotaPeriod flag from 100ms to 100µs to match the Linux CFS and k8s defaults. cpuCFSQuotaPeriod of 100ms now requires customCPUCFSQuotaPeriod flag to be set to work. (kubernetes/kubernetes#111520, @paskal) [SIG API Machinery and Node]

v25.3.0

Kubernetes API Version: v1.25.3

Feature Copy Section

• Adds support for loading CA certificates from a file using the idp-certificate-authority key for the oidc plugin. (#1916, @vgupta3)

v25.3.0b1

Kubernetes API Version: v1.25.3

Feature Copy Section

• Adds support for loading CA certificates from a file using the idp-certificate-authority key for the oidc plugin. (#1916, @vgupta3)

v25.2.0b1

Kubernetes API Version: v1.25.3

Feature Copy Section

• Adds support for loading CA certificates from a file using the idp-certificate-authority key for the oidc plugin. (#1916, @vgupta3)

v25.2.0a1

Kubernetes API Version: v1.25.2

API Change Copy Section

• Revert regression that prevented client-go latency metrics to be reported with a template URL to avoid label cardinality. (kubernetes/kubernetes#112055, @aanm) [SIG API Machinery]
• Add NodeInclusionPolicy to TopologySpreadConstraints in PodSpec. (kubernetes/kubernetes#108492, @kerthcet)
• Added KMS v2alpha1 support. (kubernetes/kubernetes#111126, @aramase)
• Added a deprecated warning for node beta label usage in PV/SC/RC and CSI Storage Capacity. (kubernetes/kubernetes#108554, @pacoxu)
• Added a new feature gate CheckpointRestore to enable support to checkpoint containers. If enabled it is possible to checkpoint a container using the newly kubelet API (/checkpoint/{podNamespace}/{podName}/{containerName}). (kubernetes/kubernetes#104907, @adrianreber) [SIG Node and Testing]
• Added alpha support for user namespaces in pods phase 1 (KEP 127, feature gate: UserNamespacesStatelessPodsSupport) (kubernetes/kubernetes#111090, @rata)
• As of v1.25, the PodSecurity restricted level no longer requires pods that set .spec.os.name="windows" to also set Linux-specific securityContext fields. If a 1.25+ cluster has unsupported out-of-skew nodes prior to v1.23 and wants to ensure namespaces enforcing the restricted policy continue to require Linux-specific securityContext fields on all pods, ensure a version of the restricted prior to v1.25 is selected by labeling the namespace (for example, pod-security.kubernetes.io/enforce-version: v1.24) (kubernetes/kubernetes#105919, @ravisantoshgudimetla)
• Changed ownership semantics of PersistentVolume's spec.claimRef from atomic to granular. (kubernetes/kubernetes#110495, @alexzielenski)
• Extended ContainerStatus CRI API to allow runtime response with container resource requests and limits that are in effect.
  • UpdateContainerResources CRI API now supports both Linux and Windows. (kubernetes/kubernetes#111645, @vinaykul)
• For v1.25, Kubernetes will be using Golang 1.19, In this PR the version is updated to 1.19rc2 as GA is not yet available. (kubernetes/kubernetes#111254, @dims)
• Introduced NodeIPAM support for multiple ClusterCIDRs (kubernetes/kubernetes#2593) as an alpha feature.
  Set feature gate MultiCIDRRangeAllocator=true, determines whether the MultiCIDRRangeAllocator controller can be used, while the kube-controller-manager flag below will pick the active controller.
  Enabled the MultiCIDRRangeAllocator by setting --cidr-allocator-type=MultiCIDRRangeAllocator flag in kube-controller-manager. (kubernetes/kubernetes#109090, @sarveshr7)
• Introduced PodHasNetwork condition for pods. (kubernetes/kubernetes#111358, @ddebroy)
• Introduced support for handling pod failures with respect to the configured pod failure policy rules. (kubernetes/kubernetes#111113, @mimowo)
• Introduction of the DisruptionTarget pod condition type. Its reason field indicates the reason for pod termination:
  • PreemptionByKubeScheduler (Pod preempted by kube-scheduler)
  • DeletionByTaintManager (Pod deleted by taint manager due to NoExecute taint)
  • EvictionByEvictionAPI (Pod evicted by Eviction API)
  • DeletionByPodGC (an orphaned Pod deleted by PodGC) (kubernetes/kubernetes#110959, @mimowo)
• Kube-Scheduler ComponentConfig is graduated to GA, kubescheduler.config.k8s.io/v1 is available now.
  Plugin SelectorSpread is removed in v1. (kubernetes/kubernetes#110534, @kerthcet)
• Local Storage Capacity Isolation feature is GA in 1.25 release. For systems (rootless) that cannot check root file system, please use kubelet config --local-storage-capacity-isolation=false to disable this feature. Once disabled, pod cannot set local ephemeral storage request/limit, and emptyDir sizeLimit niether. (kubernetes/kubernetes#111513, @jingxu97)
• Make PodSpec.Ports' description clearer on how this information is only informational and how it can be incorrect. (kubernetes/kubernetes#110564, @j4m3s-s) [SIG API Machinery, Network and Node]
• On compatible systems, a mounter's Unmount implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on umount command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point. (kubernetes/kubernetes#109676, @cartermckinnon) [SIG Storage]
• PersistentVolumeClaim objects are no longer left with storage class set to nil forever, but will be updated retroactively once any StorageClass is set or created as default. (kubernetes/kubernetes#111467, @RomanBednar)
• Promote StatefulSet minReadySeconds to GA. This means --feature-gates=StatefulSetMinReadySeconds=true are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation (kubernetes/kubernetes#110896, @ravisantoshgudimetla) [SIG API Machinery, Apps and Testing]
• Promoted CronJob's TimeZone support to beta. (kubernetes/kubernetes#111435, @soltysh)
• Promoted DaemonSet MaxSurge to GA. This means --feature-gates=DaemonSetUpdateSurge=true are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation . (kubernetes/kubernetes#111194, @ravisantoshgudimetla)
• Scheduler: included supported ScoringStrategyType list in error message for NodeResourcesFit plugin (kubernetes/kubernetes#111206, @SataQiu)
• The Go API for logging configuration in k8s.io/component-base was moved to k8s.io/component-base/logs/api/v1. The configuration file format and command line flags are the same as before. (kubernetes/kubernetes#105797, @pohly)
• The Pod spec.podOS field is promoted to GA. The IdentifyPodOS feature gate unconditionally enabled, and will no longer be accepted as a --feature-gates parameter in 1.27. (kubernetes/kubernetes#111229, @ravisantoshgudimetla)
• The PodTopologySpread is respected after rolling upgrades. (kubernetes/kubernetes#111441, @denkensk)
• The CSIInlineVolume feature has moved from beta to GA. (kubernetes/kubernetes#111258, @dobsonj)
• The PodSecurity admission plugin has graduated to GA and is enabled by default. The admission configuration version has been promoted to pod-security.admission.config.k8s.io/v1. (kubernetes/kubernetes#110459, @wangyysde)
• The endPort field in Network Policy is now promoted to GA
  Network Policy providers that support endPort field now can use it to specify a range of ports to apply a Network Policy.
  Previously, each Network Policy could only target a single port.
  Please be aware that endPort field MUST BE SUPPORTED by the Network Policy provider. In case your provider does not support endPort and this field is specified in a Network Policy, the Network Policy will be created covering only the port field (single port). (kubernetes/kubernetes#110868, @rikatz)
• The metadata.clusterName field is completely removed. This should not have any user-visible impact. (kubernetes/kubernetes#109602, @lavalamp)
• The minDomains field in Pod Topology Spread is graduated to beta (kubernetes/kubernetes#110388, @sanposhiho) [SIG API Machinery and Apps]
• The command line flag enable-taint-manager for kube-controller-manager is deprecated and will be removed in 1.26. The feature that it supports, taint based eviction, is enabled by default and will continue to be implicitly enabled when the flag is removed. (kubernetes/kubernetes#111411, @alculquicondor)
• This release added support for NodeExpandSecret for CSI driver client which enables the CSI drivers to make use of this secret while performing node expansion operation based on the user request. Previously there was no secret provided as part of the nodeexpansion call, thus CSI drivers did not make use of the same while expanding the volume at the node side. (kubernetes/kubernetes#105963, @zhucan)
• Ephemeral Containers are now generally available (GA). The EphemeralContainers feature gate is always enabled and should be removed from --feature-gates flag on the kube-apiserver and the kubelet command lines. The EphemeralContainers feature gate is deprecated and scheduled for removal in a future release. (kubernetes/kubernetes#111402, @verb)
• Introduces support for handling pod failures with respect to the configured pod failure policy rules (kubernetes/kubernetes#111113, @mimowo) [SIG API Machinery, Apps, Auth, Scheduling and Testing]
• NodeIPAM support for multiple ClusterCIDRs (https://github.com/kubernetes/enhancements/issues/2593) introduced as an alpha feature.
  Setting feature gate MultiCIDRRangeAllocator=true, determines whether the MultiCIDRRangeAllocator controller can be used, while the kube-controller-manager flag below will pick the active controller.
  Enable the MultiCIDRRangeAllocator by setting --cidr-allocator-type=MultiCIDRRangeAllocator flag in kube-controller-manager. (kubernetes/kubernetes#109090, @sarveshr7) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Instrumentation, Network and Testing]
• The CSIInlineVolume feature has moved from beta to GA. (kubernetes/kubernetes#111258, @dobsonj) [SIG API Machinery, Apps, Auth, Instrumentation, Storage and Testing]
• Added alpha support for user namespaces in pods phase 1 (KEP 127, feature gate: UserNamespacesSupport) (kubernetes/kubernetes#111090, @rata) [SIG Apps, Auth, Network, Node, Storage and Testing]
• Adds KMS v2alpha1 support (kubernetes/kubernetes#111126, @aramase) [SIG API Machinery, Auth, Instrumentation and Testing]
• As of v1.25, the PodSecurity restricted level no longer requires pods that set .spec.os.name="windows" to also set Linux-specific securityContext fields. If a 1.25+ cluster has unsupported out-of-skew nodes prior to v1.23 and wants to ensure namespaces enforcing the restricted policy continue to require Linux-specific securityContext fields on all pods, ensure a version of the restricted prior to v1.25 is selected by labeling the namespace (for example, pod-security.kubernetes.io/enforce-version: v1.24) (kubernetes/kubernetes#105919, @ravisantoshgudimetla) [SIG API Machinery, Apps, Auth, Testing and Windows]
• Changes ownership semantics of PersistentVolume's spec.claimRef from atomic to granular. (kubernetes/kubernetes#110495, @alexzielenski) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation and Testing]
• Extends ContainerStatus CRI API to allow runtime response with container resource requests and limits that are in effect.
  • UpdateContainerResources CRI API now supports both Linux and Windows.
    For details, see KEPs below. (kubernetes/kubernetes#111645, @vinaykul) [SIG Node]
• For v1.25, Kubernetes will be using golang 1.19, In this PR we update to 1.19rc2 as GA is not yet available. (kubernetes/kubernetes#111254, @dims) [SIG Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]
• Introduce PodHasNetwork condition for pods (kubernetes/kubernetes#111358, @ddebroy) [SIG Apps, Node and Testing]
• Introduction of the DisruptionTarget pod condition type. Its reason field indicates the reason for pod termination:
  • PreemptionByKubeScheduler (Pod preempted by kube-scheduler)
  • DeletionByTaintManager (Pod deleted by taint manager due to NoExecute taint)
  • EvictionByEvictionAPI (Pod evicted by Eviction API)
  • DeletionByPodGC (an orphaned Pod deleted by PodGC) (kubernetes/kubernetes#110959, @mimowo) [SIG Apps, Auth, Node, Scheduling and Testing]
• Kube-Scheduler ComponentConfig is graduated to GA, kubescheduler.config.k8s.io/v1 is available now.
  Plugin SelectorSpread is removed in v1. (kubernetes/kubernetes#110534, @kerthcet) [SIG API Machinery, Scheduling and Testing]
• Local Storage Capacity Isolation feature is GA in 1.25 release. For systems (rootless) that cannot check root file system, please use kubelet config --local-storage-capacity-isolation=false to disable this feature. Once disabled, pod cannot set local ephemeral storage request/limit, and emptyDir sizeLimit niether. (kubernetes/kubernetes#111513, @jingxu97) [SIG API Machinery, Node, Scalability and Scheduling]
• PersistentVolumeClaim objects are no longer left with storage class set to nil forever, but will be updated retroactively once any StorageClass is set or created as default. (kubernetes/kubernetes#111467, @RomanBednar) [SIG Apps, Storage and Testing]
• Promote CronJob's TimeZone support to beta (kubernetes/kubernetes#111435, @soltysh) [SIG API Machinery, Apps and Testing]
• Promote DaemonSet MaxSurge to GA. This means --feature-gates=DaemonSetUpdateSurge=true are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation (kubernetes/kubernetes#111194, @ravisantoshgudimetla) [SIG Apps]
• Respect PodTopologySpread after rolling upgrades (kubernetes/kubernetes#111441, @denkensk) [SIG API Machinery, Apps, Scheduling and Testing]
• Scheduler: include supported ScoringStrategyType list in error message for NodeResourcesFit plugin (kubernetes/kubernetes#111206, @SataQiu) [SIG Scheduling]
• The Pod spec.podOS field is promoted to GA. The IdentifyPodOS feature gate unconditionally enabled, and will no longer be accepted as a --feature-gates parameter in 1.27. (kubernetes/kubernetes#111229, @ravisantoshgudimetla) [SIG API Machinery, Apps and Windows]
• The command line flag enable-taint-manager for kube-controller-manager is deprecated and will be removed in 1.26.
  The feature that it supports, taint based eviction, is enabled by default and will continue to be implicitly enabled when the flag is removed. (kubernetes/kubernetes#111411, @alculquicondor) [SIG API Machinery]
• Ephemeral Containers are now generally available. The EphemeralContainers feature gate is always enabled and should be removed from --feature-gates flag on the kube-apiserver and the kubelet command lines. The EphemeralContainers feature gate is deprecated and scheduled for removal in a future release. (kubernetes/kubernetes#111402, @verb) [SIG API Machinery, Apps, Node, Storage and Testing]
• Added a new feature gate CheckpointRestore to enable support to checkpoint containers. If enabled it is possible to checkpoint a container using the newly kubelet API (/checkpoint/{podNamespace}/{podName}/{containerName}). (kubernetes/kubernetes#104907, @adrianreber) [SIG Node and Testing]
• EndPort field in Network Policy is now promoted to GA
  Network Policy providers that support endPort field now can use it to specify a range of ports to apply a Network Policy.
  Previously, each Network Policy could only target a single port.
  Please be aware that endPort field MUST BE SUPPORTED by the Network Policy provider. In case your provider does not support endPort and this field is specified in a Network Policy, the Network Policy will be created covering only the port field (single port). (kubernetes/kubernetes#110868, @rikatz) [SIG API Machinery, Network and Testing]
• Make PodSpec.Ports' description clearer on how this information is only informational and how it can be incorrect. (kubernetes/kubernetes#110564, @j4m3s-s) [SIG API Machinery, Network and Node]
• On compatible systems, a mounter's Unmount implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on umount command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point. (kubernetes/kubernetes#109676, @cartermckinnon) [SIG Storage]
• Promote StatefulSet minReadySeconds to GA. This means --feature-gates=StatefulSetMinReadySeconds=true are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation (kubernetes/kubernetes#110896, @ravisantoshgudimetla) [SIG API Machinery, Apps and Testing]
• The Pod spec.podOS field is promoted to GA. The IdentifyPodOS feature gate unconditionally enabled, and will no longer be accepted as a --feature-gates parameter in 1.27. (kubernetes/kubernetes#111229, @ravisantoshgudimetla) [SIG API Machinery, Apps and Windows]
• The minDomains field in Pod Topology Spread is graduated to beta (kubernetes/kubernetes#110388, @sanposhiho) [SIG API Machinery and Apps]
• The Go API for logging configuration in k8s.io/component-base was moved to k8s.io/component-base/logs/api/v1. The configuration file format and command line flags are the same as before. (kubernetes/kubernetes#105797, @pohly) [SIG API Machinery, Architecture, Cluster Lifecycle, Instrumentation, Node, Scheduling and Testing]
• The PodSecurity admission plugin has graduated to GA and is enabled by default. The admission configuration version has been promoted to pod-security.admission.config.k8s.io/v1. (kubernetes/kubernetes#110459, @wangyysde) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]
• Introduce NodeInclusionPolicies to specify nodeAffinity/nodeTaint strategy when calculating pod topology spread skew. (kubernetes/kubernetes#108492, @kerthcet) [SIG API Machinery, Apps, Scheduling and Testing]
• The metadata.clusterName field is completely removed. This should not have any user-visible impact. (kubernetes/kubernetes#109602, @lavalamp) [SIG API Machinery, Apps, Auth and Testing]
• This release add support for NodeExpandSecret for CSI driver client which enables the CSI drivers to make use of this secret while performing node expansion operation based on the user request. Previously there was no secret provided as part of the nodeexpansion call, thus CSI drivers were not make use of the same while expanding the volume at node side. (kubernetes/kubernetes#105963, @zhucan) [SIG API Machinery, Apps and Storage]