Question: return fmt.Errorf("%s: Unable to understand list result %#v (%v)", r.name, list, err) } initTrace.Step("Objects extracted") <del> r.metrics.numberOfItemsInList.Observe(float64(len(items))) </del> if err := r.syncWith(items, resourceVersion); err != nil { return fmt.Errorf("%s: Unable to sync list result: %v", r.name, err) }

What happened: After upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time. ! I use image to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. I investigated some of these kubelets with strace and pprof. With 3s of running strace I saw = 50 openat() calls to the same file from the same threadid (pid) from kubelet: If I do pprof kubelet it shows clientgo metrics and compress is taking over most of the compute time. Memory profile png: ! The reflector metrics seem to be the problem: What you expected to happen: I expect that kubelet does not need so much memory How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: One of the affected clusters has only 120 Pods and 3 Pods are in CrashLoopBackOff state, one pod since 6 days on an affected node which was investigated. Environment: Kubernetes version (use ): v1.12.5 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Linux ip-172-31-10- 4.14.63-coreos SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux Install tools: custom
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it might be a side effect of , but I couldn't find any numbers in there. The comments mention an offline discussion, was this brought up?
What happens (I only investigated it very briefly) is that kubelet creates a bunch of reflectors for every pod — secret mounts, configmap mounts, token mounts etc. Each reflector registers a number of histograms, gauges, etc. and even when the reflector is stopped and removed the metrics are never removed.

After a few days on a busy cluster you get millions of metrics bloating up kubelet and everything else using the reflectors.
This was broken since v1.12.0, so probably unrelated to .
correct that PR is only in v1.14.0-alpha.2 v1.14.0-alpha.1 and master
For everyone that finds this issue and needs a patch to disable the reflector metrics:
Do you have same metrics after disabling ReflectorMetricsProvider?
no, it drops the reflector metrics
is this indirectly caused by switching to watch-based managers?
Yeah - switching to watch resulted in more extensive using of a reflector. It seems that those metrics are directly coming from reflector (I wasn't aware of those). So it's highly probably that this change cause it. I think that we don't really need those metrics, so if we could switch them off in Kubelets, that should solve the problem.
is sent out for review to fix that. We should probably cherrypick it to previous releases if we agree that's the right way to solve it.
Actually, I realized that I don't fully understand the problem. It should be a problem in default Kubernetes (what we build from this repo), because those metrics are not switched on for kubelet: metrics factory is defaulted to noop the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.: )
Wouldn't it be better to rewrite the reflector metrics so they're aggregated in a better way rather than relying on people not accidentally enabling them? Or at least put a huge warning on top saying "this will leak memory like crazy" just so the same situation doesn't repeat in other code using client-go?
- that's a separate issue whether the metrics are reasonable or not (and node sig-node related). One thing that I would like to confirm: do you customize k8s yourself? I would like to ensure that this issue you is in your setup and not in upstream k8s.
Our production configuration is this one: Kubelet config: We run with an image called , but it's just a mirror of . So upstream Kubernetes.
I try to figure out an issue why these metrics are even there and I found issues from the past showing leaks already in older versions: Sounds for, that we need a postmortem to make this not happening again.

I don't find any issue in kubernetes nor in client-go, that could reasonable explain why these were introduced in the first place. I guess I just did not found it, and I hope someone can highlight where the decision came from.
I am running Kubernetes v1.12.3 and I don't see reflector metrics being used by kubelet ( shows nothing). FWIW, the cluster is created using kops. Any idea, how are reflector metrics enabled?
and myself will also look into this as well and see what we can find.
Thanks
The issue only exists if hyperkube is used when running kubelet, as hyperkube has imported the package for apiserver and controller-manager. If kubelet binary is used, then there're no reflector metrics (this is same with your findings). Since the metrics is not exposed by kubelet cmd, what do you think of disabling it for kubelet subcommand in hyperkube? I can't reproduce the issue (with hyperkube used) on my own cluster: ! Since both apiserver and controller-manager have also imported the package, have you also observed same issues for them?
+1
Yes, we observe the same thing for apiserver and controller-manager as well.
I would also suggest not using imports for side effects because this usually creates hard to debug and non-obvious problems like this.
So you mean disable the reflector metrics totally?
No, I suggest changing the code so that the end users (kubelet, apiserver, etc) would have to explicitly enable the metrics by calling a function instead of having this happen as a side effect of an import statement.
LGTM. WDYT? We should probably do this for both kubelet and control plane services (apiserver and controller-manager).
I'm fine with that.
+1
+1
Why wouldn't we want to just actually disable this metric wholesale? Enabling this flag would basically be the same thing as saying enable a memory leak, no?
/assign
Han points out that, since the metric has a random suffix each time, it's not very useful for monitoring anyway.
/sig api-machinery


Question: releaseServiceIP = true } <ins> nodePortOp := portallocator.StartOperation(rs.serviceNodePorts) defer nodePortOp.Finish() </ins> assignNodePorts := shouldAssignNodePorts(service) svcPortToNodePort := map[int]int{} for i := range service.Spec.Ports {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): K8S 1.5.3, and kubectl 1.5.3 Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): xenial Kernel (e.g. ): latest xenial Install tools: Others: ACS-Engine cluster, 3 masters What happened: Nodeport is assigned to dashboard and to the nginx service that was started within a minute of starting the dashboard. The dashboard is being started as part of the addon manager. What you expected to happen: Each service should receive a separate node port. How to reproduce it (as minimally and precisely as possible): Run 3 master setup, with 3 API servers, 3 addon managers, 3 controllers (but one controller is elected leader). Run dashboard as an add-on. Create service shortly after dashboard is started an assigned nodeport. We run a Jenkin's job, and this repros about 30-50% of the time. Anything else we need to know: We instrumented the build and found the following: is initially created by the addon manager by issuing a POST to/api/v1/namespaces/kube-system/services. This is where port is , and snapshotted to etcd Immediately after the POST, another addon manager issues PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard.

The post calls a release to , and this is snapshotted to etcd. API server receives the request for nginx and assigns to nginx service. all api servers start reporting message the node port for service nginx/namespace- was assigned to multiple services. Workaround: delete the nginx service, re-create the service. have you seen this before? This build has your latest changelist around , but from best we can observe is unrelated to the changes.
The attached files are the API Server Logs: Here is a timeline of events: 16:07:02.862 - - Start POST /api/v1/namespaces/kube-system/services 16:07:02.863 - - [Create() 16:07:03.173 - - allocated - in memory 16:07:03.255 - - Create] 16:07:03.107 - - bitmap.Snapshot (177), 16:07:03.256 - - Finish POST /api/v1/namespaces/kube-system/services 16:07:08.474 - - start patch k8s-dashboard 16:07:08.832 - - released from allocator bitmap, and snapshotted to etcd
because type=NodePort ? Why is this releasing the port? On purpose or bug? Ports are allocated randomly from 2767 values, and you're reliably reproducing this collision? Why did they not get the update to the service that released it? Nope. I'll try to repro.
type=NodePort ? Correct, the dashboard addon specifies is this releasing the port? On purpose or bug? Discussed with and we figured it is because the addon manager does not see dynamically created , in the addon yaml (because it is dynamically assigned). This points at a potential bug in the add-on manager that it should ignore the dynamically assigned values, ignoring the need for the patch. are allocated randomly from 2767 values, and you're reliably reproducing this collision? We are running Jenkins, and this repro's 30-50% of the time in the multi-master case. The times it does not repro is when the same API server handles both the PUT of the dashboard and the nginx services, because AllocateNext() chooses the next random value for Nginx. did they not get the update to the service that released it?

It is being released from the allocator bitmap in the registry, but not the service definition. My guess is that it is successfully being removed from the allocator bitmap, but then when is attempted to be removed from the service it fails because the service is of type NodePort. I need to instrument to confirm this. If once confirmed, suggest I look to see if the removal from service can happen first, before removing from the allocator bitmap (nodeport should be gone from service definition, before being removed from allocator registry). That way if removal from service fails, then allocator bitmap still keeps other services from locking it.
for 3) it is the same across multiple machines, I assume it's because the random number generator has the same constant seed. for 4) my guess is that on Service update we're doing the following: M1: Update to service-1 comes in without NodePort (this is an update to an existing service) M1: Allocator releases the port , writes to etcd. M1: Decides to re-allocate to the service-1, but hasn't yet written to etcd M2: New service service-2 is created without NodePort M2: Allocator also decides to allocate to service-2, writes it to etcd. M1: Allocator fails to update etcd At this point, both Service-1 and Service-2 have the same NodePort assigned to them.
And a clarification on the Addon bits: Addon manager has a service that looks like: But when there is a for it, it has in the response: so add-on manager sees a difference, and posts back something missing the which clears and resets and starts this cycle.
OK, I think I have found 1 bug in this path, but I can't convince myself it would cause this failure mode, still working on it.
The addon manager (the one we support, anyway - did you write your own?) uses , so this should be fine. The nodePort is the same as clusterIP in this regard. You can see it by repeatedly running on this and seeing the nodePort not change.
The dashboard service object does not lose port , it is only removed from etcd path /registry/ranges/servicenodeports. Also, I see the same thing using , in that the release is happening, but when I view the dashboard service object, it still shows holding .

Here's the instrumented logs: 21:50:20.7778 ] BEGIN PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard 21:50:20.8228 ] [bitmap.Release(177), 21:50:20.8228 ] bitmap.Release(177)] 21:50:20.8381 ] [bitmap.Snapshot, 21:50:20.8381 ] bitmap.Snapshot] 21:50:20.8389 ] END PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard
Can you share this patch?
ohh, maybe the same bug is triggering this! God this code is twisty. Are you game to try a patch? in , around line 435, look for change the to - just remove the , then rebuild and re-run? Tim
Yeah, its the standard addon manager. We see it continuously doing a to the API Server. (has moved to the server?)
apply generates a PATCH , Brendan Burns <:
On reflection all afternoon, I am pretty confident this is the bug. I'll send a PR, sans test probably, shortly , Tim Hockin <: We see it continuously doing a PATCH to the API Server. (has apply moved to the server?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
, Tim Hockin <:
Thanks I'm trying the patch and testing now.
Jenkins is now passing 100% on 3 master HA. Also, logs no longer show the removal of the dashboard nodeport from registry. I confirmed this for kubectl apply as well on the dashboard file. Thanks for the fix!
Thanks for the challenging bug. It reminded me how much I hate this code :) , anhowe <:


Question: return nil, err } <ins> updateService := updateServiceLoadBalancerIP(service, to.String(serviceIP)) flippedService := flipServiceInternalAnnotation(updateService) if _, err := az.reconcileLoadBalancer(clusterName, flippedService, nil, false /* wantLb */); err != nil { return nil, err } if _, err := az.reconcilePublicIP(clusterName, updateService, true /* wantLb */); err != nil { return nil, err } </ins> return lbStatus, nil }

/kind bug What happened: ILB Frontend IP configuration does not change while updating service What you expected to happen: LB Frontend IP configuration should get updated How to reproduce it (as minimally and precisely as possible): Source issue : AKS service is created using .yaml manifest. ILB is described as it is shown in the documentation: annotations: "true" spec: ports: port: 80 type: LoadBalancer loadBalancerIP: 10.240.200.2 ILB instance with the "kubernetes-internal" name is created with a correct private IP address pointing to the service described. If I change the "loadBalancerIP" to another internal IP address and re-deploy the service, the last is deployed with the correct new IP address, but "kubernetes-internal" ILB still has the old Frontend IP configuration. Environment: Cloud provider or hardware configuration: Azure
/sig azure
This also not works for public LB. /cc Could you fix the bug and cherry pick to old versions?
I got this bug today, I changed the external LoadBalancerIP on a LoadBalancer service. The service event log showed that it immediately recognised the change ("oldIP =newIP") but I had to delete/recreate the service to see the change in the Azure loadbalancer.


Question: } urlRewriter := func(targetUrl string) string { <del> return t.rewriteURL(targetUrl, req.URL) </del> <ins> return t.rewriteURL(targetUrl, req.URL, req.Host) </ins> } err := rewriteHTML(reader, writer, urlRewriter) if err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When accessing a service (e.g., ) by using the Apiserver proxy feature and the service returns with header, such as The returned header does not preserve the path if the service sent a redirect with an absolute path. Depending on the cluster, the header written by the service itself might have been and the apiserver proxy did rewrite the host:port to match the correct values. Unfortunately, it did not rewrite the path segment. What you expected to happen: I would expect the apiserver to rewrite the path segment of the response header so that it reflects the fact that it's proxied under the path: How to reproduce it (as minimally and precisely as possible): The path was not preserved. Anything else we need to know?: To work around this issue, a service may return strictly a relative url in the header such as: The feature to rewrite headers is quite common. For instance, Apache Httpd does it thanks to: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: custom_ OS (e.g. from /etc/os-release): Kernel (e.g. ):
/sig api-machinery /sig network
/assign
Thanks for the bug report!
GitHub didn't allow me to assign the following users: roycaihw. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nodeAddresses(srv) } <del> func getAddressByName(client *gophercloud.ServiceClient, name types.NodeName) (string, error) { </del> <ins> func getAddressByName(client *gophercloud.ServiceClient, name types.NodeName, needIPv6 bool) (string, error) { </ins> addrs, err := getAddressesByName(client, name) if err != nil { return "", err

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The Kubernetes master created routes in the neutron router that assign an IPv6 nexthop for a IPv4 prefix. This is not possible and causes a reachability problem between the pods. Example route: What you expected to happen: When describing a node the master has the following information It should detect the address type and choose the correct InternalIP How to reproduce it (as minimally and precisely as possible): Deploy on a Openstack cloud with IPv6 enabled using the following playbook Anything else we need to know?: The wrong next hop is coming from calling the function and nowhere is a check if the address is IPv4 or IPv6 Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ):Install tools: Others: /sig openstack
- Thank you for this detailed analysis. Seems to me that the best way to address this would be to add an IP family argument to getAddressByName() to let it know what V4/V6 family to look for for a next-hop address. In the calls to getAddressByName(), this argument could be set based on the family of route.DestinationCIDR.
Issue is probably related.


Question: # $5: pod name, which should be either etcd or etcd-events function prepare-etcd-manifest { local host_name=${ETCD_HOSTNAME:-$(hostname -s)} <del> local host_ip=$(python -c "import socket;print(socket.gethostbyname("${host_name}"))") </del> <ins> local host_ip=$(${PYTHON} -c "import socket;print(socket.gethostbyname("${host_name}"))") </ins> local etcd_cluster="" local cluster_state="new" local etcd_protocol="http"

What happened: Pick an OS image where python is python3 and the cluster will not come up. What you expected to happen: Cluster should come up. How to reproduce it (as minimally and precisely as possible): Switch the default OS image to use an OS which has python3 as the default python. Run cluster/kube- for GCE. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/assign


Question: TestSampleAPIServer(f, aggrclient, imageutils.GetE2EImage(imageutils.APIServer)) }) <ins> ginkgo.It("should manage the lifecycle of an APIService", func() { ns := f.Namespace.Name framework.Logf("ns: %v", ns) subDomain := "e2e-" + utilrand.String(5) apiServiceGroup := subDomain + ".example.com" label := map[string]string{"e2e": subDomain} labelSelector := labels.SelectorFromSet(label).String() apiServiceName := "v1alpha1." + apiServiceGroup apiServiceClient := aggrclient.ApiregistrationV1().APIServices() certCtx := setupServerCert(ns, "e2e-api") ginkgo.By(fmt.Sprintf("Create APIService %s", apiServiceName)) _, err := apiServiceClient.Create(context.TODO(), &apiregistrationv1.APIService{ ObjectMeta: metav1.ObjectMeta{ Name:   apiServiceName, Labels: label, }, Spec: apiregistrationv1.APIServiceSpec{ Service: &apiregistrationv1.ServiceReference{ Namespace: ns, Name:      "e2e-api", Port:      pointer.Int32Ptr(aggregatorServicePort), }, Group:                apiServiceGroup, Version:              "v1alpha1", CABundle:             certCtx.signingCert, GroupPriorityMinimum: 2000, VersionPriority:      200, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating apiService %s, namespace %s", apiServiceName, ns) _, err = apiServiceClient.List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "failed to list API Services") ginkgo.By("Confirm that the generated APIService has been created") err = wait.PollImmediate(apiServiceRetryPeriod, apiServiceRetryTimeout, checkApiServiceListQuantity(aggrclient, labelSelector, 1)) framework.ExpectNoError(err, "failed to count the required APIServices") ginkgo.By(fmt.Sprintf("Update status for APIService %s", apiServiceName)) var statusToUpdate, updatedStatus *apiregistrationv1.APIService updatedStatusConditions := apiregistrationv1.APIServiceCondition{ Type:    "StatusUpdate", Status:  "True", Reason:  "E2E", Message: "Set from e2e test", } err = retry.RetryOnConflict(retry.DefaultRetry, func() error { statusToUpdate, err = apiServiceClient.Get(context.TODO(), apiServiceName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve api service %s", apiServiceName) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, updatedStatusConditions) updatedStatus, err = apiServiceClient.UpdateStatus(context.TODO(), statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status. %v", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("Confirm that the generated APIService has an updated status") err = wait.PollImmediate(apiServiceRetryPeriod, apiServiceRetryTimeout, checkApiServiceStatus(aggrclient, apiServiceName, updatedStatusConditions)) framework.ExpectNoError(err, "failed to locate the required APIService status") ginkgo.By(fmt.Sprintf("Patching status for APIService %s", apiServiceName)) patchedStatusConditions := apiregistrationv1.APIServiceCondition{ Type:    "StatusPatched", Status:  "True", Reason:  "E2E", Message: "Set from e2e test", } payload := []byte(`{"status":{"conditions":[{"type":"StatusPatched","status":"True","reason":"E2E","message":"Set from e2e test"}]}}`) patchedApiService, err := apiServiceClient.Patch(context.TODO(), apiServiceName, types.MergePatchType, payload, metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Failed to patch status. %v", err) framework.Logf("Patched status conditions: %#v", patchedApiService.Status.Conditions) ginkgo.By("Confirm that the generated APIService has a patched status") err = wait.PollImmediate(apiServiceRetryPeriod, apiServiceRetryTimeout, checkApiServiceStatus(aggrclient, apiServiceName, patchedStatusConditions)) framework.ExpectNoError(err, "failed to locate the required APIService status") ginkgo.By(fmt.Sprintf("Replace APIService %s", apiServiceName)) var updatedApiService *apiregistrationv1.APIService err = retry.RetryOnConflict(retry.DefaultRetry, func() error { patchedApiService, err = apiServiceClient.Get(context.TODO(), apiServiceName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get APIService %s", apiServiceName) patchedApiService.Labels = map[string]string{ apiServiceGroup: "updated", } updatedApiService, err = apiServiceClient.Update(context.TODO(), patchedApiService, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err) framework.ExpectEqual(updatedApiService.Labels[apiServiceGroup], "updated", "updated object should have the applied label") framework.Logf("Found updated apiService label for %q", apiServiceName) ginkgo.By(fmt.Sprintf("DeleteCollection APIService %s via labelSelector: %s", apiServiceName, labelSelector)) err = aggrclient.ApiregistrationV1().APIServices().DeleteCollection(context.TODO(), metav1.DeleteOptions{GracePeriodSeconds: pointer.Int64(1)}, metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "Unable to delete apiservice %s", apiServiceName) ginkgo.By("Confirm that the generated APIService has been deleted") err = wait.PollImmediate(apiServiceRetryPeriod, apiServiceRetryTimeout, checkApiServiceListQuantity(aggrclient, labelSelector, 0)) framework.ExpectNoError(err, "failed to count the required APIServices") framework.Logf("APIService %s has been deleted.", apiServiceName) }) </ins> }) func cleanTest(client clientset.Interface, aggrclient *aggregatorclient.Clientset, namespace string) {

[X] Test approval issue: [X] Test PR: 7 July 2022 [x] Two weeks soak end date: 21 Jul 2022 [x] Test promotion According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. replaceApiregistrationV1APIServiceStatuspatchApiregistrationV1APIServiceStatusreplaceApiregistrationV1APIServicedeleteApiregistrationV1CollectionAPIService Using a number of existing e2e test practices a new has been created for 4 APIService endpoints. The e2e logs for this test are listed below. This query shows the following endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err == nil { t.Fatalf("expected error but got none") } <del> if tt.errorContains != "" { if !strings.Contains(err.Error(), tt.errorContains) { t.Errorf("expected an error saying %q, but got: %v", tt.errorContains, err) </del> <ins> expected := false if len(tt.errorContainsAnyOf) != 0 { for _, errStr := range tt.errorContainsAnyOf { if strings.Contains(err.Error(), errStr) { expected = true break } </ins> } } <ins> if !expected { t.Errorf("expected the error to be any of %q, but got: %v", tt.errorContainsAnyOf, err) } </ins> return }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: (regardlessoffailurepolicy)and_fail Testgrid link: Reason for failure: and Anything else we need to know: Spyglass: <!-- Please see the deflaking doc () for more guidance! --
/sig api-machinery /area apiserver
Can confirm this PR:
/assign
/triage accepted
The kube-apiserver processing flow is as follows, (non-critical paths are omitted) ... // other filters parse the http client timeout setting, sets the Ctx with Deadline in the http.Request ...// other filters select the http.Request.Context().Done() Pay attention to the processing logic in . Although the Context of http.Request has been cancelled, the Mutex on baseTimeoutWriter needs to be obtained between entering the function. The inner-handler may have written , ready to write , but the current code flow thinks that it should be Panic quickly, and it is recovered by and sends http2.Reset to the Client. As a result, a appears on the client side.
Currently seeing it in periodic and in presubmit for
Regardless of WithCache or WithoutCache, here is not reasonable.
/priority important-soon /milestone v1.21
PS: There were some recent failures:
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-integration per: triage link: FYI
may be one fix, may also be necessary (thanks for opening these
I believe that can effectively repair the issue, but currently is not the key PR to fix the issue.


Question: // Check builtin modules exist or not if len(modules) != 0 { <del> kernelVersionFile := "/proc/sys/kernel/osrelease" b, err := r.Executor.Command("cut", "-f1", "-d", " ", kernelVersionFile).CombinedOutput() if err != nil { errors = append(errors, fmt.Errorf("error getting os release kernel version: %v(%s)", err, out)) return nil, errors } kernelVersion := strings.TrimSpace(string(b)) </del> builtinModsFilePath := fmt.Sprintf("/lib/modules/%s/modules.builtin", kernelVersion) out, err := r.Executor.Command("cut", "-f1", "-d", " ", builtinModsFilePath).CombinedOutput() if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--kube-proxy cannot use ipvs mode on Linux 4.19 due to a check for the module, which was changed to (). uses to check for : It should instead accept on Linux 4.19 or above. What happened: kube-proxy fell back from ipvs mode to iptables mode. What you expected to happen: kube-proxy using ipvs. How to reproduce it (as minimally and precisely as possible): Launch a cluster on an image using Linux 4.19 or above with kube-proxy using ipvs mode (see ). Environment: Kubernetes version (use ): All Kernel (e.g. ): 4.19 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig network
/area ipvs /assign /cc
Okay, so should check {nfconntrackipv4 | nf_conntrack} and expect load at least one module?
I think so
Please just go ahead :)


Question: name: "scale up stabilization, reuse one of obsolete recommendation element", key:  "", recommendations: []timestampedRecommendation{ <del> {10, time.Now().Add(-100 * time.Minute)}, {6, time.Now().Add(-1 * time.Minute)}, {5, time.Now().Add(-2 * time.Minute)}, {9, time.Now().Add(-3 * time.Minute)}}, </del> <ins> {10, now.Add(-100 * time.Minute)}, {6, now.Add(-1 * time.Minute)}, {5, now.Add(-2 * time.Minute)}, {9, now.Add(-3 * time.Minute)}}, </ins> currentReplicas:              1, prenormalizedDesiredReplicas: 100, expectedStabilizedReplicas:   5, expectedRecommendations: []timestampedRecommendation{ <del> {100, time.Now()}, {6, time.Now()}, {5, time.Now()}, {9, time.Now()}, </del> <ins> {100, now}, {6, now}, {5, now}, {9, now}, </ins> }, scaleUpStabilizationWindowSeconds: 300, <ins> }, { name: "scale up and down stabilization, do not scale up when prenormalized rec goes down", key:  "", recommendations: []timestampedRecommendation{ {2, now.Add(-100 * time.Minute)}, {3, now.Add(-3 * time.Minute)}, }, currentReplicas:                     2, prenormalizedDesiredReplicas:        1, expectedStabilizedReplicas:          2, scaleUpStabilizationWindowSeconds:   300, scaleDownStabilizationWindowSeconds: 300, }, { name: "scale up and down stabilization, do not scale down when prenormalized rec goes up", key:  "", recommendations: []timestampedRecommendation{ {2, now.Add(-100 * time.Minute)}, {1, now.Add(-3 * time.Minute)}, }, currentReplicas:                     2, prenormalizedDesiredReplicas:        3, expectedStabilizedReplicas:          2, scaleUpStabilizationWindowSeconds:   300, scaleDownStabilizationWindowSeconds: 300, </ins> }, } for _, tc := range tests {

When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement.

Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)


Question: StabilityLevel: metrics.ALPHA, }, []string{"result"}) // PodScheduleSuccesses counts how many pods were scheduled. <ins> // This metric will be initialized again in Register() to assure the metric is not no-op metric. </ins> PodScheduleSuccesses = scheduleAttempts.With(metrics.Labels{"result": "scheduled"}) // PodScheduleFailures counts how many pods could not be scheduled. <ins> // This metric will be initialized again in Register() to assure the metric is not no-op metric. </ins> PodScheduleFailures = scheduleAttempts.With(metrics.Labels{"result": "unschedulable"}) // PodScheduleErrors counts how many pods could not be scheduled due to a scheduler error. <ins> // This metric will be initialized again in Register() to assure the metric is not no-op metric. </ins> PodScheduleErrors            = scheduleAttempts.With(metrics.Labels{"result": "error"}) DeprecatedSchedulingDuration = metrics.NewSummaryVec( &metrics.SummaryOpts{

What happened: two scheduler metrics, and , was not recorded in . scheduler has some code for this already. It seems to be a bug. What you expected to happen: and was recorded How to reproduce it (as minimally and precisely as possible): /king bug /sig scheduling
/assign
What do you expect for ? And I think pending pods are available in metrics, but not from scheduler. You can get it from pods info metrics.
actually, scheduler has some code already for recording these metrics. But I found a bug in these code. I will post my PR to fix this issue soon. // I updated the issue description for clarifying.


Question: // Deprecated func RawMustRegister(cs ...prometheus.Collector) { defaultRegistry.RawMustRegister(cs...) <del> // sideload global prom registry as fallback for _, c := range cs { prometheus.Register(c) } </del> } // RawRegister registers a prometheus collector but uses the global registry, this

What happened: CI failed when deprecated metrics. Refer to The reason is that a hidden metrics won't be created thus can't be registered to prometheus's registry. I guess the other PR failed for the same reason. What you expected to happen: How to reproduce it (as minimally and precisely as possible): When starting the cluster, panic happens Anything else we need to know?: In my option, we need to remove backoff from legacyregistry. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig instrumentation
Discoveries after run go test ( by ) after removing backoff and deprecating some metrics: The kubelet's metrics are protected by the unit tests, and they work well . The only block is , that is I mean register to legacy registry and gather from Prometheus.
By backoff do you mean the fallback call to prometheus.Register in the various legacyregistry register functions?
Yes. I opened a PR and see what would happen. (I guess it should be fine now)


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/test/e2e/framework" <ins> imageutils "k8s.io/kubernetes/test/utils/image" "time" </ins> "github.com/onsi/ginkgo" ) <ins> const ( podTemplateRetryPeriod  = 1 * time.Second podTemplateRetryTimeout = 1 * time.Minute ) </ins> var _ = ginkgo.Describe("[sig-node] PodTemplates", func() { f := framework.NewDefaultFramework("podtemplate") /*

[x] Original approval issue: [x] two weeks soak end date: 2 July 2020 This issue is created to allow edit by According to this APIsnoop query, there are still an endpoint which is untested. /wip /hold a set of 3 Pod Templates with a static label within a Namespace. that all 3 Pod Templates with the label in the Namespace are created. the set of Namespaced Pod Templates with a label via DeleteCollection. that all Pod Templates with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
/area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Merge 17 July 2020
Reopened this issue. <detailsIn response to : /reopen Merge 17 July 2020 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "fmt" <ins> "strings" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/errors"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): azurefile azurefile 0777 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Microsoft Azure Container Service OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Linux 4.4.0-28-generic -Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x8664 x8664 x8664 GNU/Linux What happened: I attempted to set up an Azure File Volume using the built-in volume type azureFile. Everything works, except that the files and directories defaults to permission 0777, always. When I tried to manually mount within the container, I could customise dirmode and filemode, meaning that this change should be possible. Reading the source code confirms it - dirmode=0777 and filemode=0777 seem hardcoded. What you expected to happen: Being able to supply dirmode and filemode to the mount. e.g. How to reproduce it (as minimally and precisely as possible): N/A. This is a feature request.

Anything else do we need to know: N/A
This is a big requirement, using an azure file volume for postgresql for instance fails due to permissions as initdb tries to change the permissions to be less restrictive but is unable to.
Evaluating compatibility between a certain setup of a volume mount and an arbitrary piece of software would be the responsibility of the integrator, wouldn't it? It seems a big requirement for K8s itself to have to ensure PostgreSQL can do what it wants. An integrator who tries to use Azure File Storage for PostgreSQL should know better to mount with the correct permission.
The point is that you cannot mount with permissions, they're hard coded to 0777.
Ah. Yeah. Sorry for misunderstanding the comment above. Let's say it is an "important" requirement :)
/sig azure
/assign andyzhangx
I have a for this, use mount options in storage class, below is an example:
Has this change made it into any release yet? I can't seem to find it... Or was the issue resolved in some other way? I'm facing the issue you originally reported, i.e. using a pod volume to an azureFile, and it always mounting with filemode and dirmode of 0755... and I need it to be 0777. Is there a workaround? TIA.
azure file mountOptions feature is available from v1.8.5, here is the example :
Thanks -- I was still having trouble until I the storageClassName: "" to the claim... then realized I was hitting issue which I can avoid by setting storageClassName: "default" to both the pv and pvc, and now everything works. Thanks again!


Question: // Complete assigns RenameContextOptions from the args. func (o *RenameContextOptions) Complete(cmd *cobra.Command, args []string, out io.Writer) error { if len(args) != 2 { <del> cmd.Help() return fmt.Errorf("Unexpected args: %v", args) </del> <ins> return helpErrorf(cmd, "Unexpected args: %v", args) </ins> } o.contextName = args[0]

The '-context' does not seem to add any value
add priority and team please.
Blocked on
/sig cli


Question: } // we're not writing to this stream errorStream.Close() <ins> defer pf.streamConn.RemoveStreams(errorStream) </ins> errorChan := make(chan error) go func() {

The port forwarding client leaks memory when a port forwarding session is reused over time. The port forwarder client should not leak memory. forwarding a port with kubectl. and close many connections to the forwarded port. The amount of memory leaked for each connection is quite small, so many connections are needed to observe the leak. This is related to , which was reported for the kubelet and fixed in . However, the same problem happens to the client. The client needs to call on the , otherwise the streams created by will be kept in memory until the port forwarder itself is closed, even if the streams themselves closed long before. <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted
/cc


Question: klog.Errorf("Failed to find a shared datastore matching zone %s. err: %+v", zonesToSearch, err) return "", err } <del> // Prepare error msg to be used later, if required. err = fmt.Errorf("The specified datastore %s does not match the provided zones : %s", datastoreName, zonesToSearch) } found := false // Check if the selected datastore belongs to the list of shared datastores computed. for _, sharedDs := range sharedDsList { if datastoreInfo, found = candidateDatastores[sharedDs.Info.Url]; found { klog.V(4).Infof("Datastore validation succeeded") found = true break </del> <ins> found := false for _, sharedDs := range sharedDsList { if datastoreInfo, found = candidateDatastores[sharedDs.Info.Url]; found { klog.V(4).Infof("Datastore validation succeeded") found = true break } } if !found { err = fmt.Errorf("The specified datastore %s does not match the provided zones : %s", datastoreName, zonesToSearch) klog.Error(err) return "", err </ins> } <del> } if !found { klog.Error(err) return "", err </del> } } }

Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign


Question: func (m *APIRegistrationManager) EnabledVersions() []unversioned.GroupVersion { ret := []unversioned.GroupVersion{} for _, groupMeta := range m.groupMetaMap { <del> ret = append(ret, groupMeta.GroupVersions...) </del> <ins> for _, version := range groupMeta.GroupVersions { if m.IsEnabledVersion(version) { ret = append(ret, version) } } </ins> } return ret }

introduced a bug where doesn't consult anymore... I think. I was trying to debug another problem and spotted this. I think only clients use this function. APIServer has an entirely different mechanism for tracking enabled/disabled things. cc


Question: } count++ <del> if err := info.Refresh(patchedObj, true); err != nil { return err } </del> oldData, err := json.Marshal(info.Object) if err != nil { return err

The bug is introduced in The oldData, newData in seem always to be the same. So it always showing . After digging a bit, I found that: the place where we save the may be wrong. sometimes oldData and newData are different because of different s, even though the patch is . cc:
cc
info.Object is updated by the info.Refresh call. We should compute old data prior to refreshing the info bug was introduced in
I think the second bug is because using read from stream has less infomation than when you fetch from server, so but didn't


Question: watchCacheSizes := make(map[schema.GroupResource]int) watchCacheSizes[schema.GroupResource{Resource: "replicationcontrollers"}] = maxInt(5*clusterSize, 100) watchCacheSizes[schema.GroupResource{Resource: "endpoints"}] = maxInt(10*clusterSize, 1000) <ins> watchCacheSizes[schema.GroupResource{Resource: "endpointslices", Group: "discovery.k8s.io"}] = maxInt(10*clusterSize, 1000) </ins> watchCacheSizes[schema.GroupResource{Resource: "nodes"}] = maxInt(5*clusterSize, 1000) watchCacheSizes[schema.GroupResource{Resource: "pods"}] = maxInt(50*clusterSize, 1000) watchCacheSizes[schema.GroupResource{Resource: "services"}] = maxInt(5*clusterSize, 1000)

The regression is caused by two changes: Changes in reflector () which results in issuing LIST with RV="" (meaning it will go to etcd) on WATCH restarts caused by 410 (RV too old) errors Enabling EndpointSlice API without bumping the cache size for EndpointSlices (will be fixed in ). The default cache size is 100, which is not enough for resources that can change as rapidly as EndpointSlice (to compare cache size for Endpoints, which EndpointSlices is replacing, was 10* # nodes), which results in WATCHes being terminated with 410 and then issuing heavy LISTs to restart them (see above). If you take into account that this happens on each node (kube-proxy is watching EndpointSlices) the burst in heavy LIST can effectively kill control-plane which we observe in scale tests. Below the graphs showing exactly this situation in kubemark-5K test (last two runs failed in exactly the same way): ! ! We hope that will fix the issue and make our tests passing. We're currently running the tests manually to confirm that. /sig scalability /milestone v1.17
/priority critical-urgent This is a release blocker


Question: return ret } <del> type UnstructuredNegotiatedSerializer struct{} </del> <ins> type unstructuredNegotiatedSerializer struct { typer   runtime.ObjectTyper creator runtime.ObjectCreater } </ins> <del> func (s UnstructuredNegotiatedSerializer) SupportedMediaTypes() []runtime.SerializerInfo { </del> <ins> func (s unstructuredNegotiatedSerializer) SupportedMediaTypes() []runtime.SerializerInfo { </ins> return []runtime.SerializerInfo{ { MediaType:        "application/json", EncodesAsText:    true, <del> Serializer:       json.NewSerializer(json.DefaultMetaFactory, UnstructuredCreator{}, discovery.NewUnstructuredObjectTyper(nil), false), PrettySerializer: json.NewSerializer(json.DefaultMetaFactory, UnstructuredCreator{}, discovery.NewUnstructuredObjectTyper(nil), true), </del> <ins> Serializer:       json.NewSerializer(json.DefaultMetaFactory, s.creator, s.typer, false), PrettySerializer: json.NewSerializer(json.DefaultMetaFactory, s.creator, s.typer, true), </ins> StreamSerializer: &runtime.StreamSerializerInfo{ EncodesAsText: true, <del> Serializer:    json.NewSerializer(json.DefaultMetaFactory, UnstructuredCreator{}, discovery.NewUnstructuredObjectTyper(nil), false), </del> <ins> Serializer:    json.NewSerializer(json.DefaultMetaFactory, s.creator, s.typer, false), </ins> Framer:        json.Framer, }, }, } } <del> func (s UnstructuredNegotiatedSerializer) EncoderForVersion(serializer runtime.Encoder, gv runtime.GroupVersioner) runtime.Encoder { </del> <ins> func (s unstructuredNegotiatedSerializer) EncoderForVersion(serializer runtime.Encoder, gv runtime.GroupVersioner) runtime.Encoder { </ins> return versioning.NewDefaultingCodecForScheme(Scheme, crEncoderInstance, nil, gv, nil) } <del> func (s UnstructuredNegotiatedSerializer) DecoderToVersion(serializer runtime.Decoder, gv runtime.GroupVersioner) runtime.Decoder { return unstructured.UnstructuredJSONScheme </del> <ins> func (s unstructuredNegotiatedSerializer) DecoderToVersion(serializer runtime.Decoder, gv runtime.GroupVersioner) runtime.Decoder { return unstructuredDecoder{delegate: Codecs.DecoderToVersion(serializer, gv)} } type unstructuredDecoder struct { delegate runtime.Decoder } func (d unstructuredDecoder) Decode(data []byte, defaults *schema.GroupVersionKind, into runtime.Object) (runtime.Object, *schema.GroupVersionKind, error) { // Delegate for things other than Unstructured. if _, ok := into.(runtime.Unstructured); !ok && into != nil { return d.delegate.Decode(data, defaults, into) } return unstructured.UnstructuredJSONScheme.Decode(data, defaults, into) } type unstructuredObjectTyper struct { delegate          runtime.ObjectTyper unstructuredTyper runtime.ObjectTyper } func (t unstructuredObjectTyper) ObjectKinds(obj runtime.Object) ([]schema.GroupVersionKind, bool, error) { // Delegate for things other than Unstructured. if _, ok := obj.(runtime.Unstructured); !ok { return t.delegate.ObjectKinds(obj) } return t.unstructuredTyper.ObjectKinds(obj) } func (t unstructuredObjectTyper) Recognizes(gvk schema.GroupVersionKind) bool { return t.delegate.Recognizes(gvk) || t.unstructuredTyper.Recognizes(gvk) </ins> } var crEncoderInstance = crEncoder{}

BUG REPORT Kubernetes version (use ): v1.6.2 Environment: Cloud provider or hardware configuration: GKE What happened: Create TPR "MyResource" Create namespace "soak-test" Create a "MyResource" "a" under "soak-test":Delete namespace "soak-test"wait namespace to be deleted MyResource "a" still exists after ns is deleted.
/area third-party-resource
/sig cli
I'd like to take a look at the problem if nobody else is working on this
Root Cause of this is the fact that Namespace on ThirdPartyResources is getting ignored (both passed as command --namespace, or the one present in yaml) ThirdPartyResources always end up in default namespace (which is causing confusion reported in bug)
How long has that bug existed? might related to your other findings. We know that bug doesn't exist in CRDs (we have tests specifically checking namespace handling), so I'd suggest converting to those as soon as possible.
Isn't CRD available in 1.7+ ?
Yes. If the bug has existed for multiple releases (not a regression), I don't think this is a blocker for 1.7. It could still be patched, just not a blocker for the current release.
Migration to CRD still is not automated and will still take some time. Have you considered that?
Yes, but if the behavior is not a regression, then this does not block the release. The fact that we have a clear path forward to a working resource is gravy.
This bug existed in 1.4 (), got fixed in 1.5, but appears again in 1.6 .
Ok. Not a blocker for 1.7, but if the patch is reasonable as a fix we can merge it into the service stream. For 1.7, I'd still recommend moving to CRD as quickly as possible.
I was able to reproduce this with CRD on master HEAD. What level of tests are there? Anything e2e?
most of them and specifically a storage one. You're saying we have the from above?
No, I don't see any evidence that they always end up in the default namespace (neither for TPR nor CRD). I'm not sure what that's referring to.

What I was able to reproduce is the bug described at the top of this issue: Create item in namespace, delete namespace, item remains. I observed this for TPR on and , and for CRD on .
The problem seems to be that is misspelled as in this line: If I fix that, namespace cleanup works for TPR. However, for CRD, now namespace cleanup gets stuck and controller manager logs this: Edit: To clarify, I "fixed" the check in the linked code by removing the entire highlighted block. It seems like the singular is not the only problem. It was also using the internal version of when was expected.
Regarding namespace cleanup getting stuck for CRD after the "fix", the error on the apiserver side is: According to the stack trace, the error comes from here:
yes, not only the is misspelled, but also the mismatch of the version. this is why in my original patch, I tried to ignore the version of .
I'd like to keep this open until we have a regression test since this is at least the third time we've fixed this. I'll add a test as part of making sure this fix works for CRD.


Question: if nodeID != "" { return true, nil } <del> // kubelet plugin registration service not implemented is a terminal error, no need to retry if strings.Contains(getNodeInfoError.Error(), "no handler registered for plugin type") { return false, getNodeInfoError </del> <ins> if getNodeInfoError != nil { klog.Warningf("Error calling CSI NodeGetInfo(): %v", getNodeInfoError.Error()) </ins> } // Continue with exponential backoff return false, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet dies with a panic exception if CSI plugin returns NodeId == "" and nil error. Exception log see at the end of the message What you expected to happen: Kubelet should just raise an error to the plugin and ignore the registration. How to reproduce it (as minimally and precisely as possible): Create a CSI plugin with only Node that would implement GetNodeInfo as follows: Register it as a DaemonSet and see kublet crush Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.10-gke.17 Cloud provider or hardware configuration: GKE OS (e.g: ): Container-Optimized OS (cos) Kernel (e.g. ): - Install tools: Network plugin and version (if this is a network-related bug): Others: Related exception Related line of code: - no check whether error is not nil before calling Error() on it.
/sig storage
/sig node is this the version of the kubelet too? 14 is out of support. is the issue reproducible with a 1.15+ kubelet?
/assign
I checked the code - same defect still exists in current codebase.
thanks /priority important-soon
Wow, that was fast. Thanks!


Question: rpc RemoveContainer(RemoveContainerRequest) returns (RemoveContainerResponse) {} // ListContainers lists all containers by filters. rpc ListContainers(ListContainersRequest) returns (ListContainersResponse) {} <del> // ContainerStatus returns status of the container. </del> <ins> // ContainerStatus returns status of the container. If the container is not // present, returns an error. </ins> rpc ContainerStatus(ContainerStatusRequest) returns (ContainerStatusResponse) {} // ExecSync runs a command in a container synchronously.

The behavior when requested id doesn't exist is not defined clearly in CRI. Currently, we define that should return when requested image doesn't exist (), and kubelet is relying on this behavior now (). However, we haven't clearly defined the behavior of and . Currently, they return error when requested sandbox/container doesn't exist, and kubelet is also relying on this behavior. Not super urgent, just track the issue here. /cc
/assign
Added for clarifying the behavior.


Question: validPrimaryIP = utilnet.IsIPv6String validSecondaryIP = utilnet.IsIPv4String } <del> for _, ip := range podStatus.IPs { </del> <ins> for _, ip := range podIPs { </ins> if validPrimaryIP(ip) { <del> podIPs = append(podIPs, v1.PodIP{IP: ip}) </del> <ins> ips = append(ips, ip) </ins> break } } <del> for _, ip := range podStatus.IPs { </del> <ins> for _, ip := range podIPs { </ins> if validSecondaryIP(ip) { <del> podIPs = append(podIPs, v1.PodIP{IP: ip}) </del> <ins> ips = append(ips, ip) </ins> break } } <del> apiPodStatus.PodIPs = podIPs if len(podIPs) > 0 { apiPodStatus.PodIP = podIPs[0].IP </del> <ins> return ips } // convertStatusToAPIStatus creates an api PodStatus for the given pod from // the given internal pod status.  It is purely transformative and does not // alter the kubelet state at all. func (kl *Kubelet) convertStatusToAPIStatus(pod *v1.Pod, podStatus *kubecontainer.PodStatus) *v1.PodStatus { var apiPodStatus v1.PodStatus // copy pod status IPs to avoid race conditions with PodStatus #102806 podIPs := make([]string, len(podStatus.IPs)) for j, ip := range podStatus.IPs { podIPs[j] = ip } // make podIPs order match node IP family preference #97979 podIPs = kl.sortPodIPs(podIPs) for _, ip := range podIPs { apiPodStatus.PodIPs = append(apiPodStatus.PodIPs, v1.PodIP{IP: ip}) } if len(apiPodStatus.PodIPs) > 0 { apiPodStatus.PodIP = apiPodStatus.PodIPs[0].IP </ins> } // set status for Pods created on versions of kube older than 1.6

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In a dual-stack cluster with IPv4 as default family (first kubelet IP), a container is created with a env var set to . When the container is running, properly contains the IPv4 But at runtime, this env var (used in container args) is expanded to the pod IPv6 instead of the IPv4. When using , it should always return the value actually exposed as and not something else. Kubernetes version:Cloud provider or hardware configuration: Bare metal cluster OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux 5.8.0-59-generic 66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x8664 x8664 x86_64 GNU/Linux Network plugin and version (if this is a network-related bug): cilium 1.10 Others: CRI-O 1.21 + crun.
Note: I have a configuration that matches the ones that had IP ordering issue before was fixed. So it may be related.
/sig network
Just a guess, but I wonder if it may be caused by
That's it. The fix from properly sort IP when computing the Pod status, but the sorted IPs are then replaced by computed IPs as this log shows.
/triage accepted Thanks for the detailed report
/assign


Question: total := *first + *second return &total } <ins> func makePodStorageStats(s *statsapi.PodStats, rootFsInfo *cadvisorapiv2.FsInfo, resourceAnalyzer stats.ResourceAnalyzer, hostStatsProvider HostStatsProvider, isCRIStatsProvider bool) { podNs := s.PodRef.Namespace podName := s.PodRef.Name podUID := types.UID(s.PodRef.UID) var ephemeralStats []statsapi.VolumeStats if vstats, found := resourceAnalyzer.GetPodVolumeStats(podUID); found { ephemeralStats = make([]statsapi.VolumeStats, len(vstats.EphemeralVolumes)) copy(ephemeralStats, vstats.EphemeralVolumes) s.VolumeStats = append(append([]statsapi.VolumeStats{}, vstats.EphemeralVolumes...), vstats.PersistentVolumes...) } logStats, err := hostStatsProvider.getPodLogStats(podNs, podName, podUID, rootFsInfo) if err != nil { klog.V(6).ErrorS(err, "Unable to fetch pod log stats", "pod", klog.KRef(podNs, podName)) // If people do in-place upgrade, there might be pods still using // the old log path. For those pods, no pod log stats is returned. // We should continue generating other stats in that case. // calcEphemeralStorage tolerants logStats == nil. } etcHostsStats, err := hostStatsProvider.getPodEtcHostsStats(podUID, rootFsInfo) if err != nil { klog.V(6).ErrorS(err, "Unable to fetch pod etc hosts stats", "pod", klog.KRef(podNs, podName)) } s.EphemeralStorage = calcEphemeralStorage(s.Containers, ephemeralStats, rootFsInfo, logStats, etcHostsStats, isCRIStatsProvider) } </ins>

I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes.

I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ).

The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet?

and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes.

It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is .

However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.


Question: } func TestEnsureLoadBalancerDeletesExistingExternalLb(t *testing.T) { <del> t.Parallel() </del> vals := DefaultTestClusterValues() gce, err := fakeGCECloud(vals) require.NoError(t, err)

Since merged on Friday night, my PR () has failed several retests of pull-kubernetes-bazel-test. My PR is unrelated to the load balancer so I suspect that introduced a flake. I see several runs of that test against master that aren't failing, so clearly the test works sometimes (and the original PR passed it several times to before it merged). Perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: My PR had one good run of pull-kubernetes-bazel-test before merged, but has failed several times since with no code changes to my PR. What you expected to happen: The tests should consistently pass unless related problem code is introduced.
/cc /cc /sig gcp /kind flake
My PR started failing recently too. See https://k8s-
Hey, thanks for identifying the issue. contains tests for - but the race condition was caused when tests with modified a singleton variable in a mock. The short-term fix is removing the calls, but the long-term fix is to not use a singleton. Fix is .


Question: // handle watch separately since we cannot watch multiple resource types isWatch, isWatchOnly := util.GetFlagBool(cmd, "watch"), util.GetFlagBool(cmd, "watch-only") if isWatch || isWatchOnly { <del> r := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand(cmd)). </del> <ins> r := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand()). </ins> NamespaceParam(cmdNamespace).DefaultNamespace(). SelectorParam(selector). ResourceTypeOrNameArgs(true, args...).

I think the need for this argument to the factory methods has passed - it means that callers who want to provide configurable behavior have to do it through the factory itself, which is acceptable. We can remove it whenever.
I removed most of the unused arguments. But, There are a few methods still using the argument. Namely, , , and . Before proceeding any further I wanted to ask what should be done about these methods? Deprecated/Refactored/Removed ?
The factory methods were the most important. The pattern before was people using the flags by name (cmd.Flags().Get("foo")), but now factory methods must use pointer flag vars, so there's nothing that should need the command there. The printer methods are optional units that can be used when you add the flags for them to your command. Unless we refactor the definition of those flags to use pointers and return a printer struct (something I had started a while back and never finished), it's not worth changing them right now. The other consumer of factory is OpenShift - we wrap the factory to add our additional types (implementations of things like describers, action verbs, etc): - but we will pick up and remove use of cmd when we rebase.
Thanks for the explanation. I've cleaned the as well. Print method APIs are untouched as you suggested.
cc
Looks like this was done by PR


Question: # Check to see if I can access the URL /logs/ kubectl auth can-i get /logs/ <ins> # Check to see if I can approve certificates.k8s.io kubectl auth can-i approve certificates.k8s.io </ins> # List all allowed actions in namespace "foo" kubectl auth can-i --list --namespace=foo`) <del> resourceVerbs       = sets.NewString("get", "list", "watch", "create", "update", "patch", "delete", "deletecollection", "use", "bind", "impersonate", "*") </del> <ins> resourceVerbs       = sets.NewString("get", "list", "watch", "create", "update", "patch", "delete", "deletecollection", "use", "bind", "impersonate", "*", "approve") </ins> nonResourceURLVerbs = sets.NewString("get", "put", "post", "head", "options", "delete", "patch", "*") // holds all the server-supported resources that cannot be discovered by clients. i.e. users and groups for the impersonate verb nonStandardResourceNames = sets.NewString("users", "groups")

What happened: If you use to test whether a user can approve certificate signing requests, you get a response indicating that is not a valid verb An example of the verb being used can be seen What you expected to happen: should be recognized as a valid verb on objects. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: I think that needs to be to the kubectl auth can-i code Environment: Kubernetes client and server versions (use ): Cloud provider or hardware configuration: OS (e.g: ):
This issue is currently awaiting triage. SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think is a valid verb for kubectl auth command. Valid verbs are in
so in that line of code there are a number of verbs which are only usuable on specific objects for example which works on users. AFAIK the point of that line is to inform kubectl what the valid verbs are on any object in Kubernetes, so that it knows what's possible when users use the command. the verb is valid on certificate objects as shown in the docs example I listed. Without being present in that line of code, will incorrectly state that there is no verb in Kubernetes.
I tested it, seems working when we add in that list. But the question is should we add it or not?
There have been cases where Kubectl has had additions to handle non-standard resources/verbs in the past (e.g. I raised a couple of years back). AFAIK the explicit list of verbs here is designed to address this point, it's just not got approve in it. You could ask SIG-Auth about it, I'm sure they'd have more info.
/transfer kubenertes /sig auth Decision can be made by sig-auth
Something went wrong or the destination repo kubernetes/kubenertes does not exist. <detailsIn response to : /transfer kubenertes /sig auth Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
/triage accepted
This will work even without adding verb there, adding verb just removes the warning. Submitted a PR to add that. /assign


Question: }() }) <del> ginkgo.AfterEach(func(ctx context.Context) { ginkgo.By("Clean all PVs") for nodeName, localVolumes := range allLocalVolumes { ginkgo.By(fmt.Sprintf("Cleaning up %d local volumes on node %q", len(localVolumes), nodeName)) cleanupLocalVolumes(ctx, config, localVolumes) } cleanupStorageClass(ctx, config) }) </del> ginkgo.It("should be able to process many pods and reuse local volumes", func(ctx context.Context) { var ( podsLock sync.Mutex

master-informing gce-cos-master-serial Kubernetes e2e suite.[It] [sig-storage] PersistentVolumes-local Stress with local volumes [Serial] should be able to process many pods and reuse local volumes 2023-02-15 15:12:38 +0000 UTC { failed [FAILED] persistentvolumes "local-pvt2rz2" not found In [AfterEach] at: test/e2e/storage/persistent_volumes- @ 02/21/23 18:15:38.461 } Failed multiple times: /sig storage
I think the AfterEach is racing with the test's AfterEach cleanup: Test's PV recycling go routine: We're supposed to before proceeding with cleanup. do you know if this is still a valid assumption?
/triage accepted
The recent Ginkgo changes didn't change anything regarding Go routines. If a test relies on a Go routine to have stopped, it needs to wait for that explicitly. Note that the created for an node automatically gets cancelled when the callback returns. However, that does not necessarily mean that Go routines terminate immediately.
x-post: This has been analyzed as a test issue and not a product issue, so it shouldn't block tomorrow's v1.27.0-alpha.3 release
Just bumping that this test with this same error as OP is still a common flake as of today. <img width="2768" alt="Screenshot 2023-03-20 at 3 22 23 PM" src="https://user-"
FYI, these are still flaking. Here are some failures over the last few days: !
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command.

<detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
! Just bumping that this is still flaky and failed thrice in a row recently.
The test seems to add a DeferCleanup to stop the goroutine mentioned in From the docs Is guaranteed the order and the DeferCleanup will always run before the existing AfterEach block?
I believe the order is , then (first-in-first-out), (first-in-last-out).
yes is correct. for this test that goroutine certainly adds some complexity that makes things a bit harder to reason about. assuming it's necessary and/or if you'd prefer to avoid making too many changes i'd recommend breaking the up into some s I'd be happy to submit a PR that does just that.
then the order is wrong, because the goroutine should be stopped before AfterEach, that may explain the races please be my guest, tag me and/or patrick for review
Certainly we don't need to fix our tests for us?
I know, I assumed he wanted to do it just for fun or something ... is the problem of text, you can't read between lines :)
lol! i'm happy to do it and contribute.
LOL Thanks a ton for all that you do !
PR is here: - I'm not familiar with all the k8s backslash verbs and nouns but let me know if you need me to do anything on the PR to help move it along. (and/or teach me how this works by telling me what I should do next)