Question: return RegisterPluginAsExtensions(pluginName, pluginNewFunc, "Filter") } <ins> // RegisterReservePlugin returns a function to register a Reserve Plugin to a given registry. func RegisterReservePlugin(pluginName string, pluginNewFunc runtime.PluginFactory) RegisterPluginFunc { return RegisterPluginAsExtensions(pluginName, pluginNewFunc, "Reserve") } // RegisterPermitPlugin returns a function to register a Permit Plugin to a given registry. func RegisterPermitPlugin(pluginName string, pluginNewFunc runtime.PluginFactory) RegisterPluginFunc { return RegisterPluginAsExtensions(pluginName, pluginNewFunc, "Permit") } // RegisterPreBindPlugin returns a function to register a PreBind Plugin to a given registry. func RegisterPreBindPlugin(pluginName string, pluginNewFunc runtime.PluginFactory) RegisterPluginFunc { return RegisterPluginAsExtensions(pluginName, pluginNewFunc, "PreBind") } </ins> // RegisterScorePlugin returns a function to register a Score Plugin to a given registry. func RegisterScorePlugin(pluginName string, pluginNewFunc runtime.PluginFactory, weight int32) RegisterPluginFunc { return RegisterPluginAsExtensionsWithWeight(pluginName, weight, pluginNewFunc, "Score")

This is a bug found in The scheduler fails to schedule the pod on the first try due to PV is out of sync. This may happen because the PV/PVC may be modified before the scheduler binder assumes them. However, the scheduler skips the pod update event and never try to scheduler the pod again: This is because when the reserve plugins fails, we didn't try to forget the pod. After the is called, the pod is in the state.
/sig scheduling /sig storage /assign /priority critical-urgent
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduling /sig storage /assign /priority critical-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var err error image := "gcr.io/google_containers/serve_hostname:v1.4" BeforeEach(func() { <ins> framework.SkipUnlessProviderIs("gce", "gke", "aws") </ins> if zoneCount <= 0 { zoneCount, err = getZoneCount(f.Client) Expect(err).NotTo(HaveOccurred()) } By(fmt.Sprintf("Checking for multi-zone cluster.  Zone count = %d", zoneCount)) framework.SkipUnlessAtLeast(zoneCount, 2, "Zone count is %d, only run for multi-zone clusters, skipping test") <del> framework.SkipUnlessProviderIs("gce", "gke", "aws") </del> // TODO: SkipUnlessDefaultScheduler() // Non-default schedulers might not spread }) It("should spread the pods of a service across zones", func() {

Filing separate follow up from . The issue should have been caught by failing. The suite needs a specific feature test that e.g. looks at node labels to verify multiple zones are present. cc
Yes, you make a good point. The tests currently work the other way around. If the tests detect a multi-zone cluster, they run, otherwise they skip. This makes it safe to run against any cluster, and makes test configuration much simpler. To achieve what you suggest, we will need to add a test for multi-zone that is skipped by default (i.e. in all other test jobs), and only runs on that test job (kubernetes-e2e-gce-ubernetes-lite), failing if multiple zones are not detected.
should we target this one for v1.4, or bump it to 1.5?
Moving out of v1.4 as this is a P2.
This issue was labelled only for sig/multicluster and is thus moved over to kubernetes/federation. If this does not seem to be right, please reopen this and notify us /close


Question: return nil, err } <del> oauthConfig, err := adal.NewOAuthConfig(env.ActiveDirectoryEndpoint, token.tenantID) </del> <ins> oauthConfig, err := adal.NewOAuthConfigWithAPIVersion(env.ActiveDirectoryEndpoint, token.tenantID, nil) </ins> if err != nil { return nil, fmt.Errorf("building the OAuth configuration for token refresh: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In the to acquire AAD token, because of the inclusion of the API version in the URLs for authorization, the returned token is not compatible with many AAD scenarios like OIDC and On-Behalf-Of flow. There is a long list of issues documenting this problem: The fix is pretty simple, use with set to . This incompatible behavior impacts What you expected to happen: Follow the to acquire AAD token How to reproduce it (as minimally and precisely as possible): use kubectl to get AAD token, paste token into and verify the audience claim Anything else we need to know?: Environment: Kubernetes version (use ): all versions Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
/sig cli
/sig cloud-provider
/assign


Question: # Test that it works when the live object doesn't exist output_message=$(! kubectl diff -f hack/testdata/pod.yaml) kube::test::if_has_string "${output_message}" 'test-pod' <ins> # Ensure diff only dry-runs and doesn't persist change kube::test::get_object_assert 'pod' "{{range.items}}{{ if eq ${id_field:?} "test-pod" }}found{{end}}{{end}}:" ':' </ins> kubectl apply -f hack/testdata/pod.yaml <ins> kube::test::get_object_assert 'pod' "{{range.items}}{{ if eq ${id_field:?} "test-pod" }}found{{end}}{{end}}:" 'found:' </ins> # Make sure that diffing the resource right after returns nothing (0 exit code). kubectl diff -f hack/testdata/pod.yaml

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.


Question: responses: 200: body: <del> application/json: example: !include examples/controller.json </del> <ins> example: !include examples/controller.json </ins> put: description: Update a controller body: <del> json/application: schema: !include doc/controller-schema.json example: !include examples/controller.json </del> <ins> schema: !include doc/controller-schema.json example: !include examples/controller.json </ins> delete: description: Delete a specific controller responses: 200: body: <del> application/json: example: | { "success": true } </del> <ins> example: | { "success": true } </ins> /services: get:

Starting a large data set (32gb allocated, 4gb resident) in the api server causes about 7.5gb allocations of . The tree for that shows: This appears to be due to openapi aggregation. If there is an easy fix in 1.7 it might be worth it. Otherwise this should be addressed for 1.8.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
This is all API server startup - for small clusters it's about 20-60% of startup time
Most of those are that is the library we used to convert our spec to proto. The key is this line: Do you have any idea how we can get a better space time/complexity here?
I have a PR for that now.
Fixed in
Even without aggregation, starting and serving OpenAPI docs was a huge chunk of startup time. The linked change makes it disappear.
This should not be related to aggregation at all (as far as I can tell from that tree). Thanks for the fix
Thanks, it's in gnostic master now.


Question: return nil } <ins> // selinux.SELinuxEnabled implementation for unit tests type seLinuxEnabledFunc func() bool </ins> // GetSELinux is common implementation of GetSELinuxSupport on Linux. <del> func GetSELinux(path string, mountInfoFilename string) (bool, error) { </del> <ins> func GetSELinux(path string, mountInfoFilename string, selinuxEnabled seLinuxEnabledFunc) (bool, error) { // Skip /proc/mounts parsing if SELinux is disabled. if !selinuxEnabled() { return false, nil } </ins> info, err := findMountInfo(path, mountInfoFilename) if err != nil { return false, err

On a busy system, kubelet failed to tell the container runtime that a pod's CSI volume supports SELinux and should be relabeled with pod's SELinux label. The pod then can't access the volume. Kubelet correctly detects that the CSI volume supports SELinux and tells the container runtime to relabel it. Unknown. Get a busy system and try hard? CSI volume plugin tried to detect that the volume supports SELinux during SetupAt. This check failed: Current CSI volume plugin code silently consumes the error and assumes that the volume does not support SELinux at all: Therefore, the volume is marked as mounted and given to the container runtime without the right SELinux options and the runtime does not relabel volume. <details22.0 </details<detailsNone </details<details</details<details</details<detailsCRI-O, but I think it's not really relevant. </details<details</details
/sig storage /triage accepted
We've updated kubelet in our clusters to v1.20.13 (which contains the cherry picked fix), but still observe the issue


Question: // No Tag or SHA specified, so just return what we have return true } if isTagged { <del> hostname, _ := dockerref.SplitHostname(named) // Check the RepoTags for an exact match </del> <ins> // Check the RepoTags for a match. </ins> for _, tag := range inspected.RepoTags { <del> // Deal with image with hostname specified if len(hostname) > 0 { if strings.HasSuffix(image, tag) { return true } } else { if tag == image { // We found a specific tag that we were looking for return true } </del> <ins> // An image name (without the tag/digest) can be [hostname '/'] component ['/' component]* // Because either the RepoTag or the name *may* contain the // hostname or not, we only check for the suffix match. if strings.HasSuffix(image, tag) || strings.HasSuffix(tag, image) { return true </ins> } } } if isDigested { algo := digest.Digest().Algorithm().String() sha := digest.Digest().Hex()

I cannot create pod. shows the following error shows the following message Tracing back to the source, the newly image check function from prints the message
when you run into such issue, have you tried docker pull to see if docker hub is down? Also you are using image kubernetes/pause for testing?
when did you build your cluster? was merged yesterday.
yes, docker pull worked for that image and I compared container id, it matches the sha in the kubelet log. The new fix is not in my tree yet. I'll check it out. Thank you for helping.
still not working console commit Merge: Author: Kubernetes Submit Queue <k8s-merge- Wed Aug 17 00:54:33 2016 -0700 Merge pull request from hodovska/-non-k8scontainers Automatic merge from submit-queue Add note: kubelet manages only k8s containers. Kubelet wrote log when accesing container which was not created in k8s, what could confuse users. That's why we note about it in documentation and lowered log level of the message to 5. Here is example of the message: bug Bugzilla link commit Author: Dominika Hodovska <: Thu Aug 11 10:00:13 2016 +0200 Add note: kubelet manages only k8s containers.

bug Bugzilla link 3m 58s 5 {kubelet rootfs-dev} spec.containers{web} Normal Pulling pulling image "kubernetes/pause" 3m 56s 5 {kubelet rootfs-dev} spec.containers{web} Warning Failed Failed to pull image "kubernetes/pause": image pull failed for unknown error 3m 56s 5 {kubelet rootfs-dev} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "web" with ErrImagePull: "image pull failed for unknown error" 3m 10s 13 {kubelet rootfs-dev} spec.containers{web} Normal BackOff Back-off pulling image "kubernetes/pause" 3m 10s 13 {kubelet rootfs-dev} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "web" with ImagePullBackOff: "Back-off pulling image "kubernetes/pause"" I0817 15:26:14. ] Connecting to docker on I0817 15:26:14. ] Start docker client with request timeout=2m0s I0817 15:30:36. ] Pulling image kubernetes/pause:latest without credentials I0817 15:30:38. ] Inspected image ("c80eebea8921a5e97f82fcf60da84f4de6399b9fc0ba683ceff67cd0dd319644") does not match kubernetes/pause:latest I0817 15:30:52. ] Pulling image kubernetes/pause:latest without credentials I0817 15:30:55. ] Inspected image ("c80eebea8921a5e97f82fcf60da84f4de6399b9fc0ba683ceff67cd0dd319644") does not match kubernetes/pause:latest I0817 15:31:19. ] Pulling image kubernetes/pause:latest without credentials I0817 15:31:21. ]

Inspected image ("c80eebea8921a5e97f82fcf60da84f4de6399b9fc0ba683ceff67cd0dd319644") does not match kubernetes/pause:latest
could you post the output of ?
I use to start my cluster, maybe there is a missing option
Uh...I think this is the opposite case of what PR was fixing. Uploaded to fix this.
Thanks for taking care of this!


Question: var _ error = &StatusError{} <ins> var knownReasons = map[metav1.StatusReason]struct{}{ // metav1.StatusReasonUnknown : {} metav1.StatusReasonUnauthorized:          {}, metav1.StatusReasonForbidden:             {}, metav1.StatusReasonNotFound:              {}, metav1.StatusReasonAlreadyExists:         {}, metav1.StatusReasonConflict:              {}, metav1.StatusReasonGone:                  {}, metav1.StatusReasonInvalid:               {}, metav1.StatusReasonServerTimeout:         {}, metav1.StatusReasonTimeout:               {}, metav1.StatusReasonTooManyRequests:       {}, metav1.StatusReasonBadRequest:            {}, metav1.StatusReasonMethodNotAllowed:      {}, metav1.StatusReasonNotAcceptable:         {}, metav1.StatusReasonRequestEntityTooLarge: {}, metav1.StatusReasonUnsupportedMediaType:  {}, metav1.StatusReasonInternalError:         {}, metav1.StatusReasonExpired:               {}, metav1.StatusReasonServiceUnavailable:    {}, } </ins> // Error implements the Error interface. func (e *StatusError) Error() string { return e.ErrStatus.Message

Checked a method error using when the error was caused by a validating webhook that returns a 403 response. In this particular case, the error was generated from a validating webhook implemented with controller-runtime that returns controller-runtime's response. to return true. returned false Many (but not all) of the other family of functions in suffer the same kind of problem. I have changes ready to submit in a PR if we can get agreement that this is a problem and that it should be fixed in the functions that ought to check the of an erorr value that is or wraps an Kubernetes version (use ): 19.3 for the server
/sig api-machinery
are you going to send a PR for this? The issue description looks reasonable. /triage accepted
Thanks, PR is submitted.
/assign


Question: You can also use `kubectl` command to see if the newly created k8s is working correctly. The `kubectl` binary is under the `cluster/ubuntu/binaries` directory. You can move it into your PATH. Then you can use the below command smoothly. <del> For example, use `$ kubectl get minions` to see if you get all your minion nodes comming up and ready. It may take some times for the minions be ready to use like below. </del> <ins> For example, use `$ kubectl get nodes` to see if all your minion nodes are in ready status. It may take some time for the minions ready to use like below. </ins> ``` <del> NAME                 LABELS             STATUS 10.10.103.162       <none>              Ready </del> <ins> NAME            LABELS                                 STATUS </ins> <del> 10.10.103.223       <none>              Ready </del> <ins> 10.10.103.162   kubernetes.io/hostname=10.10.103.162   Ready 10.10.103.223   kubernetes.io/hostname=10.10.103.223   Ready 10.10.103.250   kubernetes.io/hostname=10.10.103.250   Ready </ins> <del> 10.10.103.250       <none>              Ready </del> ```

This flag is deprecated and no longer has any effect.
I wonder which version does this include.Now ubuntu approach is using ubuntu 0.15.0.
This change was checked in this week, so it will be true for clusters at 0.18.0+ (once the next release is cut next week). So the next time the ubuntu documentation is updated, this change should be reflected.
Ohit means that minion node can be automatically discovered by the master node when then kubelet set ? No other configuration is needed?
Yes, the kubelet, when passed the location of the apiserver, will register itself and then post node update messages. The only part that may depend on your deployment environment is the set of addresses that the kubelet passes to the apiserver. They will be based on the name of the node, reverse resolving the name of the node, or checking for addresses from the cloud provider.
Got it.I will include this when I updates the doc and scripts. Besides, I have a little questions here. Eric told me the addon services do not have service-account now.Will 0.18.0+ have it?
Service accounts will be included as part of 0.18.0 as well.
got it
- makes sense


Question: 16: "3.3.17-0", 17: "3.4.3-0", 18: "3.4.3-0", <del> 19: "3.4.9-1", </del> <ins> 19: "3.4.13-0", </ins> 20: "3.4.13-0", 21: "3.4.13-0", }

What happened: Found while testing kubeadm 1.20 for openSUSE Kubic in openQA. The test attempted to bootstrap a simple kubernetes 1.19.4 cluster with but this test failed, complaining it could not find a etcd container with tag - etcd 3.4.9 images were removed from our registry after the recent CVE, which was resolved by 3.4.13, which was supported as of k8s 1.19.1 ( for reference) This seems to be because still references etcd 3.4.9 for clusters of version even though 3.4.13 is the correct version to be used. Thanks to for finding that. What you expected to happen: Kubeadm should use etcd v3.4.13 for all clusters version 1.19.1 and higher How to reproduce it (as minimally and precisely as possible): Run , check version of etcd. It should be , but with this bug it will be Anything else we need to know?: Environment: OS (e.g: ): NAME="openSUSE MicroOS" ID="opensuse-microos" IDLIKE="suse opensuse opensuse-tumbleweed" VERSIONID="" PRETTYNAME="openSUSE MicroOS" ANSICOLOR="0;32" CPENAME="cpe:/o:opensuse:microos:" BUGREPORTURL="" HOMEURL="" DOCUMENTATIONURL="" LOGO="distributor-logo" Kernel (e.g. ): Linux r2d2 5.9.12-1-default SMP Wed Dec 2 09:24:35 UTC 2020 () x8664 x8664 x8664 GNU/Linux /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/assign
According to , the cve is fixed in 3.4.13+ & 3.3.25+ & 3.5+.

: Fix CVE-2020- and CVE-2020-. Should we upgrade all etcd versions which has the issue in 1.19 or 1.18?
the fix for the default behavior at master and 1.20 is fine. for readers, note that you can always override the etcd image using ClusterConfiguration.Etcd.


Question: deps = [ "//cmd/kube-scheduler/app/config:go_default_library", "//cmd/kube-scheduler/app/options:go_default_library", <del> "//pkg/api/legacyscheme:go_default_library", </del> "//pkg/scheduler:go_default_library", "//pkg/scheduler/apis/config:go_default_library", "//pkg/scheduler/framework/runtime:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kube-Scheduler exposes metrics in a secure manner (TLS and authentication required) on port by default. If one does not authenticate itself, the error message displayed is: What you expected to happen: Kube-Scheduler should emit a clear and understandable error message if authentication fails. Kube-Controller-Manager does that the following way: How to reproduce it (as minimally and precisely as possible): up a Kubernetes Cluster to access the exposed metrics endpoint (https://metrics-server:/metrics) without authentication Anything else we need to know?: xref: Environment: Kubernetes version (use ): 1.18.6 Cloud provider or hardware configuration: Bare metal OS (e.g: ): Debian GNU/Linux 10 (buster) Kernel (e.g. ): 4.19.0-9-amd64 Install tools: Kubeadm (Kube-Scheduler )
/sig scheduling instrumentation
If ok, I can try this fix /assign
/assign


Question: local etcd_apiserver_creds="${ETCD_APISERVER_CREDS:-}" local etcd_extra_args="${ETCD_EXTRA_ARGS:-}" local suffix="$1" <del> local etcd_livenessprobe_port="$2" </del> <ins> local etcd_listen_metrics_port="$2" local etcdctl_certs="" </ins> if [[ -n "${INITIAL_ETCD_CLUSTER_STATE:-}" ]]; then cluster_state="${INITIAL_ETCD_CLUSTER_STATE}"

Currently, etcd livenessProbe is defined here: This means that it proves /health endpoint 3 times (default failureThreshold value) and if all of them fails, it restarts etcd server. /health endpoint returns "unhealthy" if one of the conditions is met (): there is an active Alarm (there are two kinds of alarm: NOSPACE and CORRUPT) there is no raft leader the latency of a QGET request doesn't exceed 1s The problem is that in most of those cases, etcd's restart isn't a right behavior: if there is NOSPACE alarm, the restart will not free that space if there is no raft leader, it means that it cannot talk to other etcd members, so restart will not fix them if the request latency 1s, the etcd cluster is overloaded which is bad, but after restart we will get similar amount of requests (if not more), so restart isn't a right action e.g. in we see that event-etcd gets restarted by kubelet when it's overloaded. If we do not restart etcd at that point, we will likely simply finish work to be done and heal ourselves. I think we should change livenessProbe to detect cases where restart will actually help. /cc
I think that's not necessary true, that may potentially help sometimes. But I agree that's probably pretty rare. /sig api-machinery
Do we have an analysis of why the 3 healthchecks in row failed in Was it timeout, timeout, timeout or something else? Do you say that if etcd is overloaded then restarting it will help? Wouldn't it be even more overloaded and run into restart-loop because it will never become /healthy in time again?
Sorry, typo. I tried to say that in none of those cases restart will help (including 1s latency), but in fact I wrote something else. I changed initial post.
should fix etcd such that it does not declares as unhealthy as a result of overload (while its still making a progress). Ideally the not-healthchecking traffic should be throttled / load-shed, such that healthcheck stays responsive. this is implemented, I would consider giving higher timeout (e.g. 30sec), such that HC request have higher chance to reach beginning of the processing queue and get responded.

is a usual disjoint between 2 purpuses of healthchecking: a) autohealing -by restart -where we need to be very conservative about timeouts, as cost is high. b) measuring SLO & alerting -where we want to measure customer's happiness, and we cannot consider request waiting for 15s+ seconds as healthy/happy state. The lower tolerance more represents customers needs. In practical application we should optimize etcd healthcheck for a), while b) should be based on prometheous metrics.
This is basically the difference between "livez" and "readyz". "livez" is whether the app is running (for autohealing) "readyz" is whether it's useful from customer point of view (e.g. for SLO) In the ideal world, those should be two separate handlers.
In addition to doubling the timeout we can also consider halving the probing period (from default 10s to 5s) and doubling failure thresholds (from 3 failed probes in row to 6). It would decrease the probability of etcd being restarted in case of temporarily overloaded master.
There is no option to keep using "/health" endpoint and changing timeout to something higher, as this is hardcoded in etcd: Also I think we should stop restarting etcd on NOSPACE or CORRUPT alarms. This way we will stop restarting on alarms and have ability to control timeout (and setting it to some higher value). Once etcd starts exposing some better endpoint that can be used for liveness probes, we will switch to that. WDYT?
/cc /triage accepted


Question: } uid := string(uuid.NewUUID()) <ins> klog.V(6).InfoS("Encrypting content using envelope service", "uid", uid, "key", string(dataCtx.AuthenticatedData())) </ins> resp, err := t.envelopeService.Encrypt(ctx, uid, newKey) if err != nil { return nil, fmt.Errorf("failed to encrypt DEK, error: %w", err)

Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.


Question: ) // Fit is determined by non-conflicting disk volumes. <del> scheduler.RegisterFitPredicate(predicates.NoDiskConflictPred, predicates.NoDiskConflict) </del> <ins> scheduler.RegisterFitPredicateFactory( predicates.NoDiskConflictPred, func(args scheduler.AlgorithmFactoryArgs) predicates.FitPredicate { return nil }, ) </ins> // GeneralPredicates are the predicates that are enforced by all Kubernetes components // (e.g. kubelet and all schedulers)

/sig scheduling /priority important-soon See as an example. Part of /assign
GitHub didn't allow me to assign the following users: angao. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig scheduling /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thx, I will do it soon.


Question: for url in "${urls[@]}"; do local file="${url##*/}" rm -f "${file}" <del> if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --max-time 80 --retry 6 --retry-delay 10 "${url}"; then </del> <ins> if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --max-time 300 --retry 6 --retry-delay 10 "${url}"; then </ins> echo "== Failed to download ${url}. Retrying. ==" elif [[ -n "${hash}" ]] && ! validate-hash "${file}" "${hash}"; then echo "== Hash validation of ${url} failed. Retrying. =="

We download the kube bundle from S3 with curl . But, now that the bundle is well over 1GB, 80 seconds is often not long enough on some of the smaller AWS instances (I think it's a combo of disk slowness, CPU slowness for the SSL and network slowness). PR coming which bumps it to 300 once I have tested it.


Question: import ( "fmt" <ins> "math" "math/bits" </ins> "k8s.io/apimachinery/pkg/api/resource" ) <ins> /* The Cloud Provider's volume plugins provision disks for corresponding PersistentVolumeClaims. Cloud Providers use different allocation unit for their disk sizes. AWS allows you to specify the size as an integer amount of GiB, while Portworx expects bytes for example. On AWS, if you want a volume of 1500MiB, the actual call to the AWS API should therefore be for a 2GiB disk. This file contains functions that help rounding a storage request based on a Cloud Provider's allocation unit. */ </ins> const ( // GB - GigaByte size GB = 1000 * 1000 * 1000

After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes.

Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.

Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/


Question: "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake" api "k8s.io/kubernetes/pkg/apis/core" <ins> v1defaults "k8s.io/kubernetes/pkg/apis/core/v1" </ins> "k8s.io/kubernetes/pkg/controller" kubelet "k8s.io/kubernetes/pkg/kubelet/types" <ins> utilpointer "k8s.io/utils/pointer" </ins> ) func TestIgnoresNonCreate(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --my mutating webhook calculates a hash of the , allows reinvocation; no side effects my validating webhook validates the hash of the and fails because the has changed webhook auditing indicates that my webhooks are the only ones processing the upon serializing the spec to JSON for both the mutator and validator, the only change is related to the "kube-api-access-xyz" projected volume, managed by a built-in k8s admission plugin the aforementioned plugin does not appear to mutate the reinvocation context, even through it may mutate a pod (robbing other mutators of the chance to observe the change): it appears that both my hooks see the injected volume, but the validation hook sees the injected SA volume with a "defaultMode" field set, and my mutation hook does NOT see this field set my validation hook should calculate the same hash mutating, validating webhooks are implemented atop controller-runtime v0.8.3, pod objects are being decoded into targets Kubernetes version (use ): 1.21.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.

): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
cc
interesting... defaults are applied when going from versioned → internal server decodes v1 Pod request → applies defaults → converts to internal Pod -in admission control operates on the internal Pod service account admission injects the kube-api-access volume admission is called internal Pod → converts to v1 Pod → serializes → sends to mutating webhook mutating webhook returns patch patch is applied to v1 Pod JSON resulting v1 Pod JSON is decoded → defaults applied (defaultMode is set on the api-token volume here) → converted to internal Pod webhook admission is called internal Pod → converts to v1 Pod → serializes → sends to validating webhook That means if any built-in admission plugin sets up data that would be modified by applying defaulting to the pod, the data sent to mutating webhooks will be missing those applied defaults until the first mutating webhook that returns a patch that changes the object
Possible solutions: defaults are applied to the internal → versioned object before invoking webhooks built-in admission always populates data matching default values for unset fields
As an aside, I'm not sure what you're using the hashing for, but be aware that hashing is fragile in the presence of defaults and new fields over time. A hash you add to a pod spec in one release might not match a hash computed in a different release, or with different admission in place.
/triage accepted /assign /priority backlog


Question: <ins> { "spec": { "unwantedModules": { "github.com/go-kit/kit": "lots of transitive dependencies, see https://github.com/prometheus/common/issues/255", "github.com/go-openapi/analysis": "use k8s.io/kube-openapi/pkg/validation/spec", "github.com/go-openapi/spec": "use k8s.io/kube-openapi/pkg/validation/spec instead", "github.com/go-openapi/strfmt": "use k8s.io/kube-openapi/pkg/validation/strfmt instead", "github.com/go-openapi/validate": "use k8s.io/kube-openapi/pkg/validation/validate instead", "github.com/influxdata/influxdb1-client": "", "go.mongodb.org/mongo-driver": "", "k8s.io/klog": "we have switched to klog v2, so avoid klog v1", "rsc.io/quote": "refer to #102833", "rsc.io/sampler": "refer to #102833", "github.com/hashicorp/golang-lru": "", "github.com/hashicorp/hcl": "", "github.com/json-iterator/go": "refer to #105030", "github.com/spf13/viper": "refer to #102598", "github.com/go-bindata/go-bindata ": "refer to #99829" } }, "status": { "references": [ "github.com/go-kit/kit", "github.com/hashicorp/golang-lru", "github.com/hashicorp/hcl", "github.com/json-iterator/go", "github.com/spf13/viper" ] } } </ins>

Add a file to track dependencies we want to remove or avoid picking back up. Examples: (as opposed to ) {analysis,spec,strfmt,validate} () and () () Add a presubmit that uses to do the following: if any items in the "unwanted but still present" list are no longer present and prompts to move them to the "absent and not allowed" list if any items in the "absent and not allowed" list are present and fails. Dependency updates often reintroduce problematic dependencies that were previously dropped. /area code-organization /cc
/sig architecture /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig architecture /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I may help with it. /assign I checked the code, there is already a logic to check forbidden dependencies. can draw a go mod graph
yeah, that was sort of ad-hoc, and doesn't do the ratcheting bit we want (record a dependency we have but don't want, once we no longer use it, require moving it to the forbidden list)


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2enode import ( "context" "fmt" "regexp" "strconv" "strings" "time" semver "github.com/blang/semver/v4" "github.com/google/go-cmp/cmp" "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" clientset "k8s.io/client-go/kubernetes" podutil "k8s.io/kubernetes/pkg/api/v1/pod" kubecm "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/test/e2e/feature" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutils "k8s.io/kubernetes/test/utils" imageutils "k8s.io/kubernetes/test/utils/image" ) const ( CgroupCPUPeriod            string = "/sys/fs/cgroup/cpu/cpu.cfs_period_us" CgroupCPUShares            string = "/sys/fs/cgroup/cpu/cpu.shares" CgroupCPUQuota             string = "/sys/fs/cgroup/cpu/cpu.cfs_quota_us" CgroupMemLimit             string = "/sys/fs/cgroup/memory/memory.limit_in_bytes" Cgroupv2MemLimit           string = "/sys/fs/cgroup/memory.max" Cgroupv2MemRequest         string = "/sys/fs/cgroup/memory.min" Cgroupv2CPULimit           string = "/sys/fs/cgroup/cpu.max" Cgroupv2CPURequest         string = "/sys/fs/cgroup/cpu.weight" CPUPeriod                  string = "100000" MinContainerRuntimeVersion string = "1.6.9" ) var ( podOnCgroupv2Node bool   = IsCgroup2UnifiedMode() cgroupMemLimit    string = Cgroupv2MemLimit cgroupCPULimit    string = Cgroupv2CPULimit cgroupCPURequest  string = Cgroupv2CPURequest ) type ContainerResources struct { CPUReq     string CPULim     string MemReq     string MemLim     string EphStorReq string EphStorLim string } type ContainerAllocations struct { CPUAlloc     string MemAlloc     string ephStorAlloc string } type TestContainerInfo struct { Name         string Resources    *ContainerResources Allocations  *ContainerAllocations CPUPolicy    *v1.ResourceResizeRestartPolicy MemPolicy    *v1.ResourceResizeRestartPolicy RestartCount int32 } func supportsInPlacePodVerticalScaling(ctx context.Context, f *framework.Framework) bool { node := getLocalNode(ctx, f) re := regexp.MustCompile("containerd://(.*)") match := re.FindStringSubmatch(node.Status.NodeInfo.ContainerRuntimeVersion) if len(match) != 2 { return false } // TODO(InPlacePodVerticalScaling): Update when RuntimeHandlerFeature for pod resize have been implemented if ver, verr := semver.ParseTolerant(match[1]); verr == nil { return ver.Compare(semver.MustParse(MinContainerRuntimeVersion)) >= 0 } return false } func getTestResourceInfo(tcInfo TestContainerInfo) (v1.ResourceRequirements, v1.ResourceList, []v1.ContainerResizePolicy) { var res v1.ResourceRequirements var alloc v1.ResourceList var resizePol []v1.ContainerResizePolicy if tcInfo.Resources != nil { var lim, req v1.ResourceList if tcInfo.Resources.CPULim != "" || tcInfo.Resources.MemLim != "" || tcInfo.Resources.EphStorLim != "" { lim = make(v1.ResourceList) } if tcInfo.Resources.CPUReq != "" || tcInfo.Resources.MemReq != "" || tcInfo.Resources.EphStorReq != "" { req = make(v1.ResourceList) } if tcInfo.Resources.CPULim != "" { lim[v1.ResourceCPU] = resource.MustParse(tcInfo.Resources.CPULim) } if tcInfo.Resources.MemLim != "" { lim[v1.ResourceMemory] = resource.MustParse(tcInfo.Resources.MemLim) } if tcInfo.Resources.EphStorLim != "" { lim[v1.ResourceEphemeralStorage] = resource.MustParse(tcInfo.Resources.EphStorLim) } if tcInfo.Resources.CPUReq != "" { req[v1.ResourceCPU] = resource.MustParse(tcInfo.Resources.CPUReq) } if tcInfo.Resources.MemReq != "" { req[v1.ResourceMemory] = resource.MustParse(tcInfo.Resources.MemReq) } if tcInfo.Resources.EphStorReq != "" { req[v1.ResourceEphemeralStorage] = resource.MustParse(tcInfo.Resources.EphStorReq) } res = v1.ResourceRequirements{Limits: lim, Requests: req} } if tcInfo.Allocations != nil { alloc = make(v1.ResourceList) if tcInfo.Allocations.CPUAlloc != "" { alloc[v1.ResourceCPU] = resource.MustParse(tcInfo.Allocations.CPUAlloc) } if tcInfo.Allocations.MemAlloc != "" { alloc[v1.ResourceMemory] = resource.MustParse(tcInfo.Allocations.MemAlloc) } if tcInfo.Allocations.ephStorAlloc != "" { alloc[v1.ResourceEphemeralStorage] = resource.MustParse(tcInfo.Allocations.ephStorAlloc) } } if tcInfo.CPUPolicy != nil { cpuPol := v1.ContainerResizePolicy{ResourceName: v1.ResourceCPU, RestartPolicy: *tcInfo.CPUPolicy} resizePol = append(resizePol, cpuPol) } if tcInfo.MemPolicy != nil { memPol := v1.ContainerResizePolicy{ResourceName: v1.ResourceMemory, RestartPolicy: *tcInfo.MemPolicy} resizePol = append(resizePol, memPol) } return res, alloc, resizePol } func initDefaultResizePolicy(containers []TestContainerInfo) { noRestart := v1.NotRequired setDefaultPolicy := func(ci *TestContainerInfo) { if ci.CPUPolicy == nil { ci.CPUPolicy = &noRestart } if ci.MemPolicy == nil { ci.MemPolicy = &noRestart } } for i := range containers { setDefaultPolicy(&containers[i]) } } func makeTestContainer(tcInfo TestContainerInfo) (v1.Container, v1.ContainerStatus) { cmd := "grep Cpus_allowed_list /proc/self/status | cut -f2 && sleep 1d" res, alloc, resizePol := getTestResourceInfo(tcInfo) tc := v1.Container{ Name:         tcInfo.Name, Image:        imageutils.GetE2EImage(imageutils.BusyBox), Command:      []string{"/bin/sh"}, Args:         []string{"-c", cmd}, Resources:    res, ResizePolicy: resizePol, } tcStatus := v1.ContainerStatus{ Name:               tcInfo.Name, AllocatedResources: alloc, } return tc, tcStatus } func makeTestPod(ns, name, timeStamp string, tcInfo []TestContainerInfo) *v1.Pod { var testContainers []v1.Container var podOS *v1.PodOS for _, ci := range tcInfo { tc, _ := makeTestContainer(ci) testContainers = append(testContainers, tc) } podOS = &v1.PodOS{Name: v1.Linux} pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: ns, Labels: map[string]string{ "time": timeStamp, }, }, Spec: v1.PodSpec{ OS:            podOS, Containers:    testContainers, RestartPolicy: v1.RestartPolicyOnFailure, }, } return pod } func verifyPodResizePolicy(pod *v1.Pod, tcInfo []TestContainerInfo) { ginkgo.GinkgoHelper() cMap := make(map[string]*v1.Container) for i, c := range pod.Spec.Containers { cMap[c.Name] = &pod.Spec.Containers[i] } for _, ci := range tcInfo { gomega.Expect(cMap).Should(gomega.HaveKey(ci.Name)) c := cMap[ci.Name] tc, _ := makeTestContainer(ci) gomega.Expect(tc.ResizePolicy).To(gomega.Equal(c.ResizePolicy)) } } func verifyPodResources(pod *v1.Pod, tcInfo []TestContainerInfo) { ginkgo.GinkgoHelper() cMap := make(map[string]*v1.Container) for i, c := range pod.Spec.Containers { cMap[c.Name] = &pod.Spec.Containers[i] } for _, ci := range tcInfo { gomega.Expect(cMap).Should(gomega.HaveKey(ci.Name)) c := cMap[ci.Name] tc, _ := makeTestContainer(ci) gomega.Expect(tc.Resources).To(gomega.Equal(c.Resources)) } } func verifyPodAllocations(pod *v1.Pod, tcInfo []TestContainerInfo) error { ginkgo.GinkgoHelper() cStatusMap := make(map[string]*v1.ContainerStatus) for i, c := range pod.Status.ContainerStatuses { cStatusMap[c.Name] = &pod.Status.ContainerStatuses[i] } for _, ci := range tcInfo { gomega.Expect(cStatusMap).Should(gomega.HaveKey(ci.Name)) cStatus := cStatusMap[ci.Name] if ci.Allocations == nil { if ci.Resources != nil { alloc := &ContainerAllocations{CPUAlloc: ci.Resources.CPUReq, MemAlloc: ci.Resources.MemReq} ci.Allocations = alloc defer func() { ci.Allocations = nil }() } } _, tcStatus := makeTestContainer(ci) if !cmp.Equal(cStatus.AllocatedResources, tcStatus.AllocatedResources) { return fmt.Errorf("failed to verify Pod allocations, allocated resources not equal to expected") } } return nil } func verifyPodStatusResources(pod *v1.Pod, tcInfo []TestContainerInfo) { ginkgo.GinkgoHelper() csMap := make(map[string]*v1.ContainerStatus) for i, c := range pod.Status.ContainerStatuses { csMap[c.Name] = &pod.Status.ContainerStatuses[i] } for _, ci := range tcInfo { gomega.Expect(csMap).Should(gomega.HaveKey(ci.Name)) cs := csMap[ci.Name] tc, _ := makeTestContainer(ci) gomega.Expect(tc.Resources).To(gomega.Equal(*cs.Resources)) } } func verifyPodContainersCgroupValues(ctx context.Context, f *framework.Framework, pod *v1.Pod, tcInfo []TestContainerInfo) error { ginkgo.GinkgoHelper() verifyCgroupValue := func(cName, cgPath, expectedCgValue string) error { mycmd := fmt.Sprintf("head -n 1 %s", cgPath) cgValue, _, err := e2epod.ExecCommandInContainerWithFullOutput(f, pod.Name, cName, "/bin/sh", "-c", mycmd) framework.Logf("Namespace %s Pod %s Container %s - looking for cgroup value %s in path %s", pod.Namespace, pod.Name, cName, expectedCgValue, cgPath) if err != nil { return fmt.Errorf("failed to find expected value '%s' in container cgroup '%s'", expectedCgValue, cgPath) } cgValue = strings.Trim(cgValue, "n") if cgValue != expectedCgValue { return fmt.Errorf("cgroup value '%s' not equal to expected '%s'", cgValue, expectedCgValue) } return nil } for _, ci := range tcInfo { if ci.Resources == nil { continue } tc, _ := makeTestContainer(ci) if tc.Resources.Limits != nil || tc.Resources.Requests != nil { var expectedCPUShares int64 var expectedCPULimitString, expectedMemLimitString string expectedMemLimitInBytes := tc.Resources.Limits.Memory().Value() cpuRequest := tc.Resources.Requests.Cpu() cpuLimit := tc.Resources.Limits.Cpu() if cpuRequest.IsZero() && !cpuLimit.IsZero() { expectedCPUShares = int64(kubecm.MilliCPUToShares(cpuLimit.MilliValue())) } else { expectedCPUShares = int64(kubecm.MilliCPUToShares(cpuRequest.MilliValue())) } cpuQuota := kubecm.MilliCPUToQuota(cpuLimit.MilliValue(), kubecm.QuotaPeriod) if cpuLimit.IsZero() { cpuQuota = -1 } expectedCPULimitString = strconv.FormatInt(cpuQuota, 10) expectedMemLimitString = strconv.FormatInt(expectedMemLimitInBytes, 10) if podOnCgroupv2Node { if expectedCPULimitString == "-1" { expectedCPULimitString = "max" } expectedCPULimitString = fmt.Sprintf("%s %s", expectedCPULimitString, CPUPeriod) if expectedMemLimitString == "0" { expectedMemLimitString = "max" } // convert cgroup v1 cpu.shares value to cgroup v2 cpu.weight value // https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2254-cgroup-v2#phase-1-convert-from-cgroups-v1-settings-to-v2 expectedCPUShares = int64(1 + ((expectedCPUShares-2)*9999)/262142) } if expectedMemLimitString != "0" { err := verifyCgroupValue(ci.Name, cgroupMemLimit, expectedMemLimitString) if err != nil { return err } } err := verifyCgroupValue(ci.Name, cgroupCPULimit, expectedCPULimitString) if err != nil { return err } err = verifyCgroupValue(ci.Name, cgroupCPURequest, strconv.FormatInt(expectedCPUShares, 10)) if err != nil { return err } } } return nil } func waitForContainerRestart(ctx context.Context, f *framework.Framework, podClient *e2epod.PodClient, pod *v1.Pod, expectedContainers []TestContainerInfo) error { ginkgo.GinkgoHelper() var restartContainersExpected []string for _, ci := range expectedContainers { if ci.RestartCount > 0 { restartContainersExpected = append(restartContainersExpected, ci.Name) } } if len(restartContainersExpected) == 0 { return nil } pod, err := podClient.Get(ctx, pod.Name, metav1.GetOptions{}) if err != nil { return err } restartedContainersCount := 0 for _, cName := range restartContainersExpected { cs, _ := podutil.GetContainerStatus(pod.Status.ContainerStatuses, cName) if cs.RestartCount < 1 { break } restartedContainersCount++ } if restartedContainersCount == len(restartContainersExpected) { return nil } if restartedContainersCount > len(restartContainersExpected) { return fmt.Errorf("more container restarts than expected") } else { return fmt.Errorf("less container restarts than expected") } } func waitForPodResizeActuation(ctx context.Context, f *framework.Framework, c clientset.Interface, podClient *e2epod.PodClient, pod, patchedPod *v1.Pod, expectedContainers []TestContainerInfo) *v1.Pod { ginkgo.GinkgoHelper() var resizedPod *v1.Pod var pErr error timeouts := framework.NewTimeoutContext() // Wait for container restart gomega.Eventually(ctx, waitForContainerRestart, timeouts.PodStartShort, timeouts.Poll). WithArguments(f, podClient, pod, expectedContainers). ShouldNot(gomega.HaveOccurred(), "failed waiting for expected container restart") // Verify Pod Containers Cgroup Values gomega.Eventually(ctx, verifyPodContainersCgroupValues, timeouts.PodStartShort, timeouts.Poll). WithArguments(f, patchedPod, expectedContainers). ShouldNot(gomega.HaveOccurred(), "failed to verify container cgroup values to match expected") // Wait for pod resource allocations to equal expected values after resize gomega.Eventually(ctx, func() error { resizedPod, pErr = podClient.Get(ctx, pod.Name, metav1.GetOptions{}) if pErr != nil { return pErr } return verifyPodAllocations(resizedPod, expectedContainers) }, timeouts.PodStartShort, timeouts.Poll). ShouldNot(gomega.HaveOccurred(), "timed out waiting for pod resource allocation values to match expected") return resizedPod } func doPodResizeTests() { f := framework.NewDefaultFramework("pod-resize-test") var podClient *e2epod.PodClient ginkgo.BeforeEach(func() { podClient = e2epod.NewPodClient(f) }) type testCase struct { name        string containers  []TestContainerInfo patchString string expected    []TestContainerInfo } noRestart := v1.NotRequired doRestart := v1.RestartContainer tests := []testCase{ { name: "Guaranteed QoS pod, one container - increase CPU & memory", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "100m", MemReq: "200Mi", MemLim: "200Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"200m","memory":"400Mi"},"limits":{"cpu":"200m","memory":"400Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "200m", MemReq: "400Mi", MemLim: "400Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, }, { name: "Guaranteed QoS pod, one container - decrease CPU & memory", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "300m", CPULim: "300m", MemReq: "500Mi", MemLim: "500Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"100m","memory":"250Mi"},"limits":{"cpu":"100m","memory":"250Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "100m", MemReq: "250Mi", MemLim: "250Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, }, { name: "Guaranteed QoS pod, one container - increase CPU & decrease memory", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "100m", MemReq: "200Mi", MemLim: "200Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"200m","memory":"100Mi"},"limits":{"cpu":"200m","memory":"100Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "200m", MemReq: "100Mi", MemLim: "100Mi"}, }, }, }, { name: "Guaranteed QoS pod, one container - decrease CPU & increase memory", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "100m", MemReq: "200Mi", MemLim: "200Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"50m","memory":"300Mi"},"limits":{"cpu":"50m","memory":"300Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "50m", CPULim: "50m", MemReq: "300Mi", MemLim: "300Mi"}, }, }, }, { name: "Guaranteed QoS pod, three containers (c1, c2, c3) - increase: CPU (c1,c3), memory (c2) ; decrease: CPU (c2), memory (c1,c3)", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "100m", MemReq: "100Mi", MemLim: "100Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:      "c2", Resources: &ContainerResources{CPUReq: "200m", CPULim: "200m", MemReq: "200Mi", MemLim: "200Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:      "c3", Resources: &ContainerResources{CPUReq: "300m", CPULim: "300m", MemReq: "300Mi", MemLim: "300Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"140m","memory":"50Mi"},"limits":{"cpu":"140m","memory":"50Mi"}}}, {"name":"c2", "resources":{"requests":{"cpu":"150m","memory":"240Mi"},"limits":{"cpu":"150m","memory":"240Mi"}}}, {"name":"c3", "resources":{"requests":{"cpu":"340m","memory":"250Mi"},"limits":{"cpu":"340m","memory":"250Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "140m", CPULim: "140m", MemReq: "50Mi", MemLim: "50Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:      "c2", Resources: &ContainerResources{CPUReq: "150m", CPULim: "150m", MemReq: "240Mi", MemLim: "240Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:      "c3", Resources: &ContainerResources{CPUReq: "340m", CPULim: "340m", MemReq: "250Mi", MemLim: "250Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory requests only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"200Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "200Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory limits only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"limits":{"memory":"400Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "400Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase memory requests only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"300Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "300Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase memory limits only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"limits":{"memory":"600Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "600Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"100m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU limits only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"limits":{"cpu":"300m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "300m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU requests only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"150m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "150m", CPULim: "200m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU limits only", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"limits":{"cpu":"500m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "500m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests and limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"100m"},"limits":{"cpu":"200m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU requests and limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"200m"},"limits":{"cpu":"400m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests and increase CPU limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"100m"},"limits":{"cpu":"500m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "500m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU requests and decrease CPU limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "400m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"200m"},"limits":{"cpu":"300m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "300m", MemReq: "250Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory requests and limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"100Mi"},"limits":{"memory":"300Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "100Mi", MemLim: "300Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase memory requests and limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"500Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "300Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory requests and increase memory limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"100Mi"},"limits":{"memory":"500Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "100Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase memory requests and decrease memory limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"300Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "300Mi", MemLim: "300Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests and increase memory limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"100m"},"limits":{"memory":"500Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "400m", MemReq: "200Mi", MemLim: "500Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU requests and decrease memory limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "400m", MemReq: "200Mi", MemLim: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"200m"},"limits":{"memory":"400Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory requests and increase CPU limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"100Mi"},"limits":{"cpu":"300m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "300m", MemReq: "100Mi", MemLim: "400Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests + limits - increase memory requests and decrease CPU limits", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "400m", MemReq: "200Mi", MemLim: "400Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"300Mi"},"limits":{"cpu":"300m"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", CPULim: "300m", MemReq: "300Mi", MemLim: "400Mi"}, }, }, }, { name: "Burstable QoS pod, one container with cpu & memory requests - decrease memory request", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", MemReq: "500Mi"}, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"400Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "200m", MemReq: "400Mi"}, }, }, }, { name: "Guaranteed QoS pod, one container - increase CPU (NotRequired) & memory (RestartContainer)", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "100m", MemReq: "200Mi", MemLim: "200Mi"}, CPUPolicy: &noRestart, MemPolicy: &doRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"200m","memory":"400Mi"},"limits":{"cpu":"200m","memory":"400Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:         "c1", Resources:    &ContainerResources{CPUReq: "200m", CPULim: "200m", MemReq: "400Mi", MemLim: "400Mi"}, CPUPolicy:    &noRestart, MemPolicy:    &doRestart, RestartCount: 1, }, }, }, { name: "Burstable QoS pod, one container - decrease CPU (RestartContainer) & memory (NotRequired)", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "200Mi", MemLim: "400Mi"}, CPUPolicy: &doRestart, MemPolicy: &noRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"50m","memory":"100Mi"},"limits":{"cpu":"100m","memory":"200Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:         "c1", Resources:    &ContainerResources{CPUReq: "50m", CPULim: "100m", MemReq: "100Mi", MemLim: "200Mi"}, CPUPolicy:    &doRestart, MemPolicy:    &noRestart, RestartCount: 1, }, }, }, { name: "Burstable QoS pod, three containers - increase c1 resources, no change for c2, decrease c3 resources (no net change for pod)", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "100Mi", MemLim: "200Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:      "c2", Resources: &ContainerResources{CPUReq: "200m", CPULim: "300m", MemReq: "200Mi", MemLim: "300Mi"}, CPUPolicy: &noRestart, MemPolicy: &doRestart, }, { Name:      "c3", Resources: &ContainerResources{CPUReq: "300m", CPULim: "400m", MemReq: "300Mi", MemLim: "400Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"150m","memory":"150Mi"},"limits":{"cpu":"250m","memory":"250Mi"}}}, {"name":"c3", "resources":{"requests":{"cpu":"250m","memory":"250Mi"},"limits":{"cpu":"350m","memory":"350Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "150m", CPULim: "250m", MemReq: "150Mi", MemLim: "250Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:      "c2", Resources: &ContainerResources{CPUReq: "200m", CPULim: "300m", MemReq: "200Mi", MemLim: "300Mi"}, CPUPolicy: &noRestart, MemPolicy: &doRestart, }, { Name:      "c3", Resources: &ContainerResources{CPUReq: "250m", CPULim: "350m", MemReq: "250Mi", MemLim: "350Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, }, { name: "Burstable QoS pod, three containers - decrease c1 resources, increase c2 resources, no change for c3 (net increase for pod)", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "100Mi", MemLim: "200Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:      "c2", Resources: &ContainerResources{CPUReq: "200m", CPULim: "300m", MemReq: "200Mi", MemLim: "300Mi"}, CPUPolicy: &noRestart, MemPolicy: &doRestart, }, { Name:      "c3", Resources: &ContainerResources{CPUReq: "300m", CPULim: "400m", MemReq: "300Mi", MemLim: "400Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"cpu":"50m","memory":"50Mi"},"limits":{"cpu":"150m","memory":"150Mi"}}}, {"name":"c2", "resources":{"requests":{"cpu":"350m","memory":"350Mi"},"limits":{"cpu":"450m","memory":"450Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "50m", CPULim: "150m", MemReq: "50Mi", MemLim: "150Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, { Name:         "c2", Resources:    &ContainerResources{CPUReq: "350m", CPULim: "450m", MemReq: "350Mi", MemLim: "450Mi"}, CPUPolicy:    &noRestart, MemPolicy:    &doRestart, RestartCount: 1, }, { Name:      "c3", Resources: &ContainerResources{CPUReq: "300m", CPULim: "400m", MemReq: "300Mi", MemLim: "400Mi"}, CPUPolicy: &noRestart, MemPolicy: &noRestart, }, }, }, { name: "Burstable QoS pod, three containers - no change for c1, increase c2 resources, decrease c3 (net decrease for pod)", containers: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "100Mi", MemLim: "200Mi"}, CPUPolicy: &doRestart, MemPolicy: &doRestart, }, { Name:      "c2", Resources: &ContainerResources{CPUReq: "200m", CPULim: "300m", MemReq: "200Mi", MemLim: "300Mi"}, CPUPolicy: &doRestart, MemPolicy: &noRestart, }, { Name:      "c3", Resources: &ContainerResources{CPUReq: "300m", CPULim: "400m", MemReq: "300Mi", MemLim: "400Mi"}, CPUPolicy: &noRestart, MemPolicy: &doRestart, }, }, patchString: `{"spec":{"containers":[ {"name":"c2", "resources":{"requests":{"cpu":"250m","memory":"250Mi"},"limits":{"cpu":"350m","memory":"350Mi"}}}, {"name":"c3", "resources":{"requests":{"cpu":"100m","memory":"100Mi"},"limits":{"cpu":"200m","memory":"200Mi"}}} ]}}`, expected: []TestContainerInfo{ { Name:      "c1", Resources: &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "100Mi", MemLim: "200Mi"}, CPUPolicy: &doRestart, MemPolicy: &doRestart, }, { Name:         "c2", Resources:    &ContainerResources{CPUReq: "250m", CPULim: "350m", MemReq: "250Mi", MemLim: "350Mi"}, CPUPolicy:    &noRestart, MemPolicy:    &noRestart, RestartCount: 1, }, { Name:         "c3", Resources:    &ContainerResources{CPUReq: "100m", CPULim: "200m", MemReq: "100Mi", MemLim: "200Mi"}, CPUPolicy:    &doRestart, MemPolicy:    &doRestart, RestartCount: 1, }, }, }, } timeouts := framework.NewTimeoutContext() for idx := range tests { tc := tests[idx] ginkgo.It(tc.name, func(ctx context.Context) { ginkgo.By("waiting for the node to be ready", func() { if !supportsInPlacePodVerticalScaling(ctx, f) || framework.NodeOSDistroIs("windows") || isRunningOnArm64() { e2eskipper.Skipf("runtime does not support InPlacePodVerticalScaling -- skipping") } }) var testPod *v1.Pod var patchedPod *v1.Pod var pErr error tStamp := strconv.Itoa(time.Now().Nanosecond()) initDefaultResizePolicy(tc.containers) initDefaultResizePolicy(tc.expected) testPod = makeTestPod(f.Namespace.Name, "testpod", tStamp, tc.containers) testPod = e2epod.MustMixinRestrictedPodSecurity(testPod) ginkgo.By("creating pod") newPod := podClient.CreateSync(ctx, testPod) ginkgo.By("verifying initial pod resources, allocations are as expected") verifyPodResources(newPod, tc.containers) ginkgo.By("verifying initial pod resize policy is as expected") verifyPodResizePolicy(newPod, tc.containers) err := e2epod.WaitForPodCondition(ctx, f.ClientSet, newPod.Namespace, newPod.Name, "Ready", timeouts.PodStartShort, testutils.PodRunningReady) framework.ExpectNoError(err, "pod %s/%s did not go running", newPod.Namespace, newPod.Name) framework.Logf("pod %s/%s running", newPod.Namespace, newPod.Name) ginkgo.By("verifying initial pod status resources") verifyPodStatusResources(newPod, tc.containers) ginkgo.By("patching pod for resize") patchedPod, pErr = f.ClientSet.CoreV1().Pods(newPod.Namespace).Patch(ctx, newPod.Name, types.StrategicMergePatchType, []byte(tc.patchString), metav1.PatchOptions{}) framework.ExpectNoError(pErr, "failed to patch pod for resize") ginkgo.By("verifying pod patched for resize") verifyPodResources(patchedPod, tc.expected) gomega.Eventually(ctx, verifyPodAllocations, timeouts.PodStartShort, timeouts.Poll). WithArguments(patchedPod, tc.containers). Should(gomega.BeNil(), "failed to verify Pod allocations for patchedPod") ginkgo.By("waiting for resize to be actuated") resizedPod := waitForPodResizeActuation(ctx, f, f.ClientSet, podClient, newPod, patchedPod, tc.expected) ginkgo.By("verifying pod resources after resize") verifyPodResources(resizedPod, tc.expected) ginkgo.By("verifying pod allocations after resize") gomega.Eventually(ctx, verifyPodAllocations, timeouts.PodStartShort, timeouts.Poll). WithArguments(resizedPod, tc.expected). Should(gomega.BeNil(), "failed to verify Pod allocations for resizedPod") ginkgo.By("deleting pod") deletePodSyncByName(ctx, f, newPod.Name) // we need to wait for all containers to really be gone so cpumanager reconcile loop will not rewrite the cpu_manager_state. // this is in turn needed because we will have an unavoidable (in the current framework) race with the // reconcile loop which will make our attempt to delete the state file and to restore the old config go haywire waitForAllContainerRemoval(ctx, newPod.Name, newPod.Namespace) }) } } func doPodResizeErrorTests() { f := framework.NewDefaultFramework("pod-resize-errors") var podClient *e2epod.PodClient ginkgo.BeforeEach(func() { podClient = e2epod.NewPodClient(f) }) type testCase struct { name        string containers  []TestContainerInfo patchString string patchError  string expected    []TestContainerInfo } tests := []testCase{ { name: "BestEffort pod - try requesting memory, expect error", containers: []TestContainerInfo{ { Name: "c1", }, }, patchString: `{"spec":{"containers":[ {"name":"c1", "resources":{"requests":{"memory":"400Mi"}}} ]}}`, patchError: "Pod QoS is immutable", expected: []TestContainerInfo{ { Name: "c1", }, }, }, } timeouts := framework.NewTimeoutContext() for idx := range tests { tc := tests[idx] ginkgo.It(tc.name, func(ctx context.Context) { ginkgo.By("waiting for the node to be ready", func() { if !supportsInPlacePodVerticalScaling(ctx, f) || framework.NodeOSDistroIs("windows") { e2eskipper.Skipf("runtime does not support InPlacePodVerticalScaling -- skipping") } }) var testPod, patchedPod *v1.Pod var pErr error tStamp := strconv.Itoa(time.Now().Nanosecond()) initDefaultResizePolicy(tc.containers) initDefaultResizePolicy(tc.expected) testPod = makeTestPod(f.Namespace.Name, "testpod", tStamp, tc.containers) testPod = e2epod.MustMixinRestrictedPodSecurity(testPod) ginkgo.By("creating pod") newPod := podClient.CreateSync(ctx, testPod) perr := e2epod.WaitForPodCondition(ctx, f.ClientSet, newPod.Namespace, newPod.Name, "Ready", timeouts.PodStartSlow, testutils.PodRunningReady) framework.ExpectNoError(perr, "pod %s/%s did not go running", newPod.Namespace, newPod.Name) framework.Logf("pod %s/%s running", newPod.Namespace, newPod.Name) ginkgo.By("verifying initial pod resources, allocations, and policy are as expected") verifyPodResources(newPod, tc.containers) verifyPodResizePolicy(newPod, tc.containers) ginkgo.By("verifying initial pod status resources and cgroup config are as expected") verifyPodStatusResources(newPod, tc.containers) ginkgo.By("patching pod for resize") patchedPod, pErr = f.ClientSet.CoreV1().Pods(newPod.Namespace).Patch(ctx, newPod.Name, types.StrategicMergePatchType, []byte(tc.patchString), metav1.PatchOptions{}) if tc.patchError == "" { framework.ExpectNoError(pErr, "failed to patch pod for resize") } else { gomega.Expect(pErr).To(gomega.HaveOccurred(), tc.patchError) patchedPod = newPod } ginkgo.By("verifying pod resources after patch") verifyPodResources(patchedPod, tc.expected) ginkgo.By("verifying pod allocations after patch") gomega.Eventually(ctx, verifyPodAllocations, timeouts.PodStartShort, timeouts.Poll). WithArguments(patchedPod, tc.expected). Should(gomega.BeNil(), "failed to verify Pod allocations for patchedPod") deletePodSyncByName(ctx, f, newPod.Name) // we need to wait for all containers to really be gone so cpumanager reconcile loop will not rewrite the cpu_manager_state. // this is in turn needed because we will have an unavoidable (in the current framework) race with the // reconcile loop which will make our attempt to delete the state file and to restore the old config go haywire waitForAllContainerRemoval(ctx, newPod.Name, newPod.Namespace) }) } } // NOTE: Pod resize scheduler resource quota tests are out of scope in e2e_node tests, //       because in e2e_node tests //          a) scheduler and controller manager is not running by the Node e2e //          b) api-server in services doesn't start with --enable-admission-plugins=ResourceQuota //             and is not possible to start it from TEST_ARGS //       Above tests are performed by doSheduletTests() and doPodResizeResourceQuotaTests() //       in test/node/pod_resize_test.go var _ = SIGDescribe("Pod InPlace Resize Container", framework.WithSerial(), feature.InPlacePodVerticalScaling, func() { if !podOnCgroupv2Node { cgroupMemLimit = CgroupMemLimit cgroupCPULimit = CgroupCPUQuota cgroupCPURequest = CgroupCPUShares } doPodResizeTests() doPodResizeErrorTests() }) </ins>

One of the review comments for PR requested to move current pod resize E2E test from test/e2e/node to test/e2enode. Please see comment: I have a lot of trust in the current E2E test after it recently failed upon rebase and CI had switched to cgroup v2 (support for it was planned for beta) - this means the test works quite well and its execution gives me high confidence. So, I believe we should keep the current E2E test as it is for now, add a new E2E test under e2enode, and perhaps eventually remove the current E2E test. The tests under e2e/node are being deprecated. So we need to add tests under e2enode which uses a different framework that has its advantages.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/status in-progress
You must be a member of the GitHub team to add status labels. If you believe you should be able to issue the /status command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.
Does this test make sense in the cluster suite too, or only in the node suite? If it's applicable to both, you should move it to instead.
Thanks good point , will check it out and update commit.

Hi, code as it is now i think is better suited to be only in the node suite. what do you think?


Question:
PLEASE NOTE: This document applies to the HEAD of the source tree
 If you are using a released version of Kubernetes, you should refer to the docs that go with that version. Documentation for other releases can be found at [releases.k8s.io](http://releases.k8s.io).  --   # Kubernetes Release Notes [djmm@google.com](mailto:djmm@google.com)
 Last Updated: 2016-3-25  - [Kubernetes Release Notes](#kubernetes-release-notes) - [Objective](#objective) - [Background](#background) - [The Problem](#the-problem) - [The (general) Solution](#the-general-solution) - [Then why not just list *every* change that was submitted, CHANGELOG-style?](#then-why-not-just-list-every-change-that-was-submitted-changelog-style) - [Options](#options) - [Collection Design](#collection-design) - [Publishing Design](#publishing-design) - [Location](#location) - [Layout](#layout) - [Alpha/Beta/Patch Releases](#alphabetapatch-releases) - [Major/Minor Releases](#majorminor-releases) - [Work estimates](#work-estimates) - [Caveats / Considerations](#caveats--considerations)  ## Objective Define a process and design tooling for collecting, arranging and publishing release notes for Kubernetes releases, automating as much of the process as possible. The goal is to introduce minor changes to the development workflow in a way that is mostly frictionless and allows for the capture of release notes as PRs are submitted to the repository. This direct association of release notes to PRs captures the intention of release visibility of the PR at the point an idea is submitted upstream. The release notes can then be more easily collected and published when the release is ready. ## Background ### The Problem Release notes are often an afterthought and clarifying and finalizing them is often left until the very last minute at the time the release is made. This is usually long after the feature or bug fix was added and is no longer on the mind of the author.  Worse, the collecting and summarizing of the release is often left to those who may know little or nothing about these individual changes! Writing and editing release notes at the end of the cycle can be a rushed, interrupt-driven and often stressful process resulting in incomplete, inconsistent release notes often with errors and omissions. ### The (general) Solution Like most things in the development/release pipeline, the earlier you do it, the easier it is for everyone and the better the outcome.  Gather your release notes earlier in the development cycle, at the time the features and fixes are added. #### Then why not just list *every* change that was submitted, CHANGELOG-style? On larger projects like Kubernetes, showing every single change (PR) would mean hundreds of entries.  The goal is to highlight the major changes for a release. ## Options 1. Use of pre-commit and other local git hooks * Experiments here using `prepare-commit-msg` and `commit-msg` git hook files were promising but less than optimal due to the fact that they would require input/confirmation with each commit and there may be multiple commits in a push and eventual PR. 1. Use of [github templates](https://github.com/blog/2111-issue-and-pull-request-templates) * Templates provide a great way to pre-fill PR comments, but there are no server-side hooks available to parse and/or easily check the contents of those templates to ensure that checkboxes were checked or forms were filled in. 1. Use of labels enforced by mungers/bots * We already make great use of mungers/bots to manage labels on PRs and it fits very nicely in the existing workflow ## Collection Design The munger/bot option fits most cleanly into the existing workflow. The design will include: 1. New labels added to github: `release-note-none`, maybe others for new release note categories - see Layout section below 1. A [new munger](https://github.com/kubernetes/kubernetes/issues/23409) that will: * Initiate a `release-note-needed` label on all new PRs * Block merge by the submit queue on all PRs labeled as `release-note-needed` * Auto-remove `release-note-needed` when one of the release-note-* labels is added * Special case for cherry-picked/branch PRs, release-note-none is not allowed ## Publishing Design ### Location With v1.2.0, the release notes were moved from their previous [github releases](https://github.com/kubernetes/kubernetes/releases) location to [CHANGELOG.md](../../CHANGELOG.md).  Going forward this seems like a good plan. Other projects do similarly. The kuberntes.tar.gz download link is also displayed along with the release notes in [CHANGELOG.md](../../CHANGELOG.md). Is there any reason to continue publishing anything to github releases if the complete release story is published in [CHANGELOG.md](../../CHANGELOG.md)? ### Layout Different types of releases will generally have different requirements in terms of layout.  As expected, major releases like v1.2.0 are going to require much more detail than the automated release notes will provide. The idea is that these mechanisms will provide 100% of the release note content for alpha, beta and most minor releases and bootstrap the content with a release note 'template' for the authors of major releases like v1.2.0. The authors can then collaborate and edit the higher level sections of the release notes in a PR, updating [CHANGELOG.md](../../CHANGELOG.md) as needed. v1.2.0 demonstrated the need, at least for major releases like v1.2.0, for several sections in the published release notes. In order to provide a basic layout for release notes in the future, new releases can bootstrap [CHANGELOG.md](../../CHANGELOG.md) with the following template types: #### Alpha/Beta/Patch Releases These are automatically generated from `release-note*` labels, but can be modified as needed. ``` Action Required * PR titles from the release-note-action-required label Other notable changes * PR titles from the release-note label ``` #### Major/Minor Releases ``` Major Themes * Add to or delete this section Other notable improvements * Add to or delete this section Experimental Features * Add to or delete this section Action Required * PR titles from the release-note-action-required label Known Issues * Add to or delete this section Provider-specific Notes * Add to or delete this section Other notable changes * PR titles from the release-note label ``` ## Work estimates * The [new munger](https://github.com/kubernetes/kubernetes/issues/23409) * Owner: @eparis * Time estimate: Mostly done * Updates to the tool that collects, organizes, publishes and sends release notifications. * Owner: @david-mcmahon * Time estimate: A few days ## Caveats / Considerations * As part of the planning and development workflow how can we capture release notes for bigger features? [#23070](https://github.com/kubernetes/kubernetes/issues/23070) * For now contributors should simply use the first PR that enables a new feature by default.  We'll revisit if this does not work well.  [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/release-notes.md?pixel)]()

Currently, PRs having label is automatically included as part of the release notes. This process does not work for big features though. We need to track issues that represent features as well and include a gist from those issues in the release notes.
From email to kubernetes-dev: One thing that was clear is that most people didn't know what should go into the release notes. In a number of cases, I had to ask specific people to add specific items to specific sections. The only way I could do that is I knew what was in the release and how the release notes should be organized. We need a template for release notes, guidelines for what belongs in release notes, and an example set of release notes (1.2.0). We also need to capture items that belong in release notes before the PRs merge. David M. has a proposal to automate release-note collection. We need to extend it with the categories we've created and require every PR assign a category (including something like release-note/none-minor-fix) before being merged by the submit queue. PRs that belong in the release notes will need to have extractable release note text in their PR descriptions. Writing release notes should be to the documented developer workflow.
For now the workaround here is to ensure that the first PR that enables a feature contain the appropriate label.
Closing. Moved to release repo.