Question: } }) <ins> ginkgo.It("should manage the lifecycle of an event", func() { eventTestName := "event-test" ginkgo.By("creating a test event") // create a test event in test namespace _, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).Create(context.TODO(), &v1.Event{ ObjectMeta: metav1.ObjectMeta{ Name: eventTestName, Labels: map[string]string{ "testevent-constant": "true", }, }, Message: "This is a test event", Reason:  "Test", Type:    "Normal", Count:   1, InvolvedObject: v1.ObjectReference{ Namespace: f.Namespace.Name, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create test event") ginkgo.By("listing all events in all namespaces") // get a list of Events in all namespaces to ensure endpoint coverage eventsList, err := f.ClientSet.CoreV1().Events("").List(context.TODO(), metav1.ListOptions{ LabelSelector: "testevent-constant=true", }) framework.ExpectNoError(err, "failed list all events") foundCreatedEvent := false var eventCreatedName string for _, val := range eventsList.Items { if val.ObjectMeta.Name == eventTestName && val.ObjectMeta.Namespace == f.Namespace.Name { foundCreatedEvent = true eventCreatedName = val.ObjectMeta.Name break } } if !foundCreatedEvent { framework.Failf("unable to find test event %s in namespace %s, full list of events is %+v", eventTestName, f.Namespace.Name, eventsList.Items) } ginkgo.By("patching the test event") // patch the event's message eventPatchMessage := "This is a test event - patched" eventPatch, err := json.Marshal(map[string]interface{}{ "message": eventPatchMessage, }) framework.ExpectNoError(err, "failed to marshal the patch JSON payload") _, err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Patch(context.TODO(), eventTestName, types.StrategicMergePatchType, []byte(eventPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch the test event") ginkgo.By("fetching the test event") // get event by name event, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).Get(context.TODO(), eventCreatedName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch the test event") framework.ExpectEqual(event.Message, eventPatchMessage, "test event message does not match patch message") ginkgo.By("updating the test event") testEvent, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).Get(context.TODO(), event.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get test event") testEvent.Series = &v1.EventSeries{ Count:            100, LastObservedTime: metav1.MicroTime{Time: time.Unix(1505828956, 0)}, } // clear ResourceVersion and ManagedFields which are set by control-plane testEvent.ObjectMeta.ResourceVersion = "" testEvent.ObjectMeta.ManagedFields = nil _, err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Update(context.TODO(), testEvent, metav1.UpdateOptions{}) framework.ExpectNoError(err, "failed to update the test event") ginkgo.By("getting the test event") event, err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Get(context.TODO(), testEvent.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get test event") // clear ResourceVersion and ManagedFields which are set by control-plane event.ObjectMeta.ResourceVersion = "" event.ObjectMeta.ManagedFields = nil if !apiequality.Semantic.DeepEqual(testEvent, event) { framework.Failf("test event wasn't properly updated: %v", diff.ObjectReflectDiff(testEvent, event)) } ginkgo.By("deleting the test event") // delete original event err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Delete(context.TODO(), eventCreatedName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete the test event") ginkgo.By("listing all events in all namespaces") // get a list of Events list namespace eventsList, err = f.ClientSet.CoreV1().Events("").List(context.TODO(), metav1.ListOptions{ LabelSelector: "testevent-constant=true", }) framework.ExpectNoError(err, "fail to list all events") foundCreatedEvent = false for _, val := range eventsList.Items { if val.ObjectMeta.Name == eventTestName && val.ObjectMeta.Namespace == f.Namespace.Name { foundCreatedEvent = true break } } if foundCreatedEvent { framework.Failf("Should not have found test event %s in namespace %s, full list of events %+v", eventTestName, f.Namespace.Name, eventsList.Items) } }) </ins> /* Release: v1.20 Testname: Event, delete a collection

[X] test approval issue : [X] test pr : [x] two weeks soak end date : 15 Jul 2022 [x] test promotion pr : According to this APIsnoop query, there are still a single remaining Event endpoint which is untested. replaceCoreV1NamespacedEvent Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the event endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "fmt" <del> "strings" </del> "sync" "time"

In the guestbook example, the redis slaves are instructed to connect to the redis-master using the proxy provided by the minion on which they reside. The minion is designated by its hostname "kubernetes-minion-N". The container cannot resolve this name to an IP address and so slave connections fail. 1) create vagrant cluster 2) create redis-master pod 3) create redis-master service 4) create redis-slave replicationController Verify redis-slave pod locations: list pods ID Image(s) Host Labels Status redis-master-2 dockerfile/redis kubernetes-minion-1/10.245.2.2 name=redis-master Running -3839-11e4-a922- brendanburns/redis-slave kubernetes-minion-1/10.245.2.2 name=redisslave,replicationController=redisSlaveController Running -3839-11e4-a922- brendanburns/redis-slave kubernetes-minion-3/10.245.2.4 name=redisslave,replicationController=redisSlaveController Running Log onto a minion containing a slave (note: vagrant hostnames are minion-N, not kubernetes-minion-N) Find the slave container docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES brendanburns/redis-slave:latest /bin/sh -c About an hour ago Up About an hour k8s--slave.---3839-11e4-a922- docker logs | tail [7] 09 Sep 17:27:08.428  Connecting to MASTER kubernetes-minion-1: [7] 09 Sep 17:27:08.430 # Unable to connect to MASTER: No such file or directory [7] 09 Sep 17:27:09.435  Connecting to MASTER kubernetes-minion-1: sudo nsenter -m -u -n -i -p -t $(docker inspect --format '{{ .State.Pid }}' ) /bin/bash [ root ]$ ping kubernetes-minion-1 ping: unknown host kubernetes-minion-1 [ root ]$ cat /etc/hosts 244.1.4 -3839-11e4-a922- 127.0.0.

1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters [ root ]$ cat nameserver 10.0.2.3 cat /proc/1/environ HOME=/rootPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=-3839-11e4-a922-0800279696e1REDISMASTERSERVICEPORT=10000REDISMASTERPORT=tcp://kubernetes-minion-1:10000REDISMASTERPORT6379TCP=tcp://kubernetes-minion-1:10000REDISMASTERPORT6379TCPPROTO=tcpREDISMASTERPORT6379TCPPORT=10000REDISMASTERPORT6379TCPADDR=kubernetes-minion-1SERVICE_HOST=kubernetes-
Tim - are you the right guy to handle this? I think we need to resolve the name of the service host and only give IPs to the minion in env variables. We can't assume the containers have the same DNS set up as the host system.
To be clear -- this example is for the checked in Vagrant stuff? If that is the case would be the guy? I'm trying to catch up on IRC discussions from earlier today.
I can look at this, but there are two possible solutions. The first solution is in the vagrant cluster (or any other local vm setup) we could run a dns name server on the kubernetes-master that can resolve each minion by host name. We would then pass a -dns option to the docker daemon to specify that name server to use in each container. The second solution is that we pass IP address instead of host name and make no assumption or override on the containers specified name server. Is there any obvious negative to this solution? Has much thought been done on the expected Nameserver used by containers spawned by our service? - if you can weigh in, I would be happy to resolve using either approach. My preference is solution two.
So, the vagrant case is a little special as there is no sane DNS set up by default. Another suggestion (3) would be to revert the stuff and just devolve to using ip addresses directly. I would do (3) before (2) before (1).

But I'm cool with any of them. As for the nameserver of the containers that are spawned -- docker currently just takes the of the host and copies that into the container. It doesn't do that for . that is the core of the issue here. If/when we do offer some level of split horizon DNS (I'd love to) we can start exposing an extra layer of DNS on top of whatever is configured for the host. That is part of the "ip per service portal" stuff that is being talked about a lot right now.
If I do (3) there is a piece of code in pod storage that mangles my hostname where I return 10.245.x.x and it asks the cloud provider to resolve the IP address for "10". Since other providers appear to also be returning IP as host, I will look to change this so kubernetes makes no modifications to hostnames that it's given. Look for PR tomorrow.
you're right I'd missed that point. This really only applies to the Vagrant cluster and so a fix to the vagrant setup is sufficient. I also agree with your priorities. Injecting a line into /etc/hosts (if that's reasonable) is the simplest solution. Either reverting or allowing a switch to pass the minion public IP rather than hostname is next. Adding a DNS service to the vagrant master and forcing DNS resolution to the master host is most complicated and least preferable. Thanks both of you for helping me think this through better. Mark
I don't grok what the difference between 2 and 3 is? I sent a PR to docker a while back () to insert arbitrary lines to /etc/hosts. I still think it is a reasonable thing to want to do.
- difference between (2) and (3) is that in the past, the vagrant cloud provider was returning the IP address as the literal hostname for the minion. I had changed this after noticing that it had some ugly side-effects in the CLI, where a call to list pods would report pod ip addresses as IP / nil.

The issue (which I now believe is a bug that I want to change independent of what we do here) is that the code at this line mangles the hostname returned by the cloudprovider: This makes using the IP address as a hostname (10.245.2.2) to have the ugly side-effect of the code then asking the cloudprovider for the IP address of hostname "10", and then the cloudprovider having no idea how to proceed. Independent of what is done in this issue, I want to submit a PR tomorrow to stop that code from manipulating the reported hostname that was provided by the cloudprovider before asking that same cloudprovider to return the IP address using a potentially different input value. so with that backlog (3) means revert vagrant cloudprovider to use IPAddress as hostname, which means when we pass in what we believe is hostname in this environment, it will just work because it is in-fact the IPAddress. (2) means do not pass in hostname but pass in IPAddress by changing more core k8s code. all that being said, there is an option (4) that I am intrigued by, - after following your PR through its final conclusion, it looks like as of docker 1.2.0 you can change the /etc/hosts file of a running container. this introduces an option where the kubelet injects the /etc/hosts file of minion into the running container so it has the /etc/hosts values of the minion. I would need to play with that support a little more to know pros/cons more. I think longer term, we need a plan on what nameserver our containers are expecting to reference, and if they are inherently different in some way. I have run into a number of times the issue where doing things like running docker in docker causes the nested container to default to the nameserver 8.8.8.8, which happens to be blocked on the Red Hat network ;-) For now to unblock I will probably pursue (3) and fix the hostname bug referenced here at the same time. It does make the CLI uglier when working with this environment, but not a huge deal, at least it will function.
Thanks both for the complete explanation of what's happening and for the suggested fixes.

I think in the long run using hostnames in SERVICEHOST and ensuriing either through IP injection into container /etc/hosts or by providing DNS are the best options. I'm not sure if there might still be cases where resolution of SERVICEHOST would still be best done by presenting the IP address or not.


Question: <ins> /var/lib/docker/containers/*/*-json.log { rotate 5 copytruncate missingok notifempty compress maxsize 10M daily create 0644 root root } </ins>

We have rotation of logs from docker containers, but not for logs from docker Without rotation, the logs may fill the root partition entirely.
I can add rotation (5 files, 100MB each) if you decide it goes for v1.0 (for me it's a blocker). I'm not sure which platforms are affected. For sure GCE. cc
Sure, send a PR and I'm sure we can merge it.


Question: otherKubelet := kubelet.NewIntegrationTestKubelet(machineList[1], &fakeDocker2) go util.Forever(func() { otherKubelet.Run(cfg2.Updates()) }, 0) go util.Forever(func() { <del> kubelet.ListenAndServeKubeletServer(otherKubelet, cfg2.Channel("http"), http.DefaultServeMux, "localhost", 10251) </del> <ins> kubelet.ListenAndServeKubeletServer(otherKubelet, cfg2.Channel("http"), "localhost", 10251) </ins> }, 0) return apiServer.URL

The Kubelet REST end-point has turned into a complicated switch-case code block that is harder to maintain overtime. The Kubelet should move to a http.ServeMux pattern and register handler functions to a pattern.
I am working on a PR to submit in next couple of days that will make this change, so feel free to assign to me if possible.
Tagging because I asked for this in one of his PRs
This one is also under my radar. If no one had time, feel free to assign it to me too.
I have a branch in progress. Feel free to offer feedback on it when I submit. Sent from my iPhone


Question: "command": [ "/bin/sh", "-c", <del> "/usr/local/bin/etcd --name etcd-{{ host_name }} --listen-peer-urls http://{{ host_name }}:{{ server_port }} --initial-advertise-peer-urls http://{{ host_name }}:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} --data-dir /var/etcd/data{{ suffix }} --initial-cluster {{ etcd_cluster }} --initial-cluster-state {{ cluster_state }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </del> <ins> "/usr/local/bin/etcd --listen-peer-urls http://127.0.0.1:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} --data-dir /var/etcd/data{{ suffix }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </ins> ], "livenessProbe": { "httpGet": {

test grid: Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it for debs: The corresponding RPM job is also faster now:


Question: You should soon be able to see all your nodes by running the `kubectl get nodes` command. Otherwise, you will need to manually create node objects. <del> ### Logging **TODO** talk about starting Logging. ### Monitoring **TODO** talk about starting Monitoring. ### DNS **TODO** talk about starting DNS. </del> <ins> ### Starting Cluster Services You will want to complete your Kubernetes clusters by adding cluster-wide services.  These are sometimes called *addons*, and [an overview of their purpose is in the admin guide]( ../../docs/admin/cluster-components.md#addons). Notes for setting up each cluster service are given below: * Cluster DNS: * required for many kubernetes examples * [Setup instructions](http://releases.k8s.io/HEAD/cluster/addons/dns/) * [Admin Guide](../admin/dns.md) * Cluster-level Logging * Multiple implementations with different storage backends and UIs. * [Elasticsearch Backend Setup Instructions](http://releases.k8s.io/HEAD/cluster/addons/fluentd-elasticsearch/) * [Google Cloud Logging Backend Setup Instructions](http://releases.k8s.io/HEAD/cluster/addons/fluentd-gcp/). * Both require running fluentd on each node. * [User Guide](../user-guide/logging.md) * Container Resource Monitoring * [Setup instructions](http://releases.k8s.io/HEAD/cluster/addons/cluster-monitoring/) * GUI * [Setup instructions](http://releases.k8s.io/HEAD/cluster/addons/kube-ui/) cluster. </ins> ## Troubleshooting

has TODOs for how to bring up cluster addons like monitoring and logging. We should fill these in. requested by user:


Question: import ( "errors" <ins> "fmt" </ins> "os" <ins> "os/exec" </ins> "runtime" <ins> "strings" </ins> "sync" "time" "unsafe"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The windows kubelet does not set .status.nodeInfo.systemUUID on the node object. What you expected to happen: The windows kubelet sets the systemUUID from How to reproduce it (as minimally and precisely as possible): Start the windows kubelet and check for UUID: Anything else we need to know?: This will affect users running the out-of-tree vsphere CCM since that depends on the systemUUID, unlike the in-tree version. Also, looks like cAdvisor is the component responsible for providing the systemUUID so let me know if this issue needs to be moved to that repo. Environment: Kubernetes version (use ): Verified on 1.13.1 and 1.14 Cloud provider or hardware configuration: None OS (e.g: ): Windows Server 2019
/sig windows /sig vmware
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: // Now apply kube reserved and system reserved limits if required. if nc.EnforceNodeAllocatable.Has(kubetypes.SystemReservedEnforcementKey) { klog.V(2).Infof("Enforcing System reserved on cgroup %q with limits: %+v", nc.SystemReservedCgroupName, nc.SystemReserved) <del> if err := enforceExistingCgroup(cm.cgroupManager, ParseCgroupfsToCgroupName(nc.SystemReservedCgroupName), nc.SystemReserved); err != nil { </del> <ins> if err := enforceExistingCgroup(cm.cgroupManager, cm.cgroupManager.CgroupName(nc.SystemReservedCgroupName), nc.SystemReserved); err != nil { </ins> message := fmt.Sprintf("Failed to enforce System Reserved Cgroup Limits on %q: %v", nc.SystemReservedCgroupName, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) return fmt.Errorf(message)

start kubelet with below config: with below config: check the cgroup, get the following output: then, start kubelet.service, run as follows: why? Environment: Kubernetes version (use ): v1.14.1 Cloud provider or hardware configuration: OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g. ): SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/area kubelet /sig node
From cgroupManagerImpl#Exists : Can you turn on log V4 so that we can see the above log ? thanks
It seems if we change return value type to error, the caller would be able to incorporate error message from Exists func.
Thanks for sharing :) Hope we can help you debug! Is the information here
thanks for your rely. when I turn on log V4, logs as followed: Cgroup path is not expected: , so when I modify , it‘s ok. so Is this normal?
Ah, it looks like its appending automatically to the cgroup name. At the very least, I think the documentation could be clearer to indicate this will happen. An additional option could be adding the suffix only if doesn't already exist.
thank you
no problem :) If this issue is resolved, would you mind closing it? Thanks!
It might be worth considering adding a little code somewhere that simply strips off the ".slice" from the cgroup name supplied in configuration option and just normalizing the input in the configuration package itself (possibly with a warning that prints out stating cgroup should not end in ".slice")
I think that's a great idea. I will work on that, unless you want to :) /assign
thank you


Question: return pod } <ins> func newHostPathTypeTestPodWithCommand(nodeSelector map[string]string, hostDir, mountDir string, hostPathType *v1.HostPathType, command string) *v1.Pod { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "test-hostpath-type-", }, Spec: v1.PodSpec{ NodeSelector:  nodeSelector, RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:  "host-path-sh-testing", Image: imageutils.GetE2EImage(imageutils.Agnhost), VolumeMounts: []v1.VolumeMount{ { Name:      "host", MountPath: mountDir, ReadOnly:  false, }, }, Command: []string{"sh"}, Args:    []string{"-c", command}, }, }, Volumes: []v1.Volume{ { Name: "host", VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ Path: hostDir, Type: hostPathType, }, }, }, }, }, } return pod } </ins> func verifyPodHostPathTypeFailure(f *framework.Framework, nodeSelector map[string]string, hostDir, pattern string, hostPathType *v1.HostPathType) { pod := newHostPathTypeTestPod(nodeSelector, hostDir, "/mnt/test", hostPathType) ginkgo.By(fmt.Sprintf("Creating pod %s", pod.Name))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Once we switched over the default runtime to containerd, the slow ci and pull jobs started timing out. The tests, which used to take 13 minutes, jumped to 30 minutes. This increase in test time is only observed in the slow job, so I suspect the slow job is exercising some functionality that may not be exercised as frequently in the other jobs. What you expected to happen: No performance regression How to reproduce it (as minimally and precisely as possible): Run the tests in the slow job with containerd Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
do we know what tests were regressing?
I did some analysis in , but basically, no single test stood out in terms of increasing the time took. I suspect that all the test cases started taking a few seconds longer, and the sum of all of contributed to the increase in job time.
I was mainly eyeballing the graphs, but we probably should get the raw data and do some more detailed analysis on it
do we have scale runs with the [slow] tests?
As an experiment, I changed the runtime back to docker in , and the slow job tests went back down to 13 minutes.
pointed out that may be relevant
/cc karan /cc
pointed out that cri-tools is not used in the kubelet path, so that particular issue may not be the cause. But maybe we can investigate image pulling as a first start. FWIW, there are at least 100+ test cases running in this job that create the same container on the same node multiple times: hostpathtype, multivolume, container restart
I think I found it. All the HostPath Socket tests are consistently never completing.
something might related
The test logs don't seem to show up in the artifacts. Running it manually, I see it hung on the exec during BeforeEach(): However when I do this manually using "kubectl exec" it is fine.

found a work around
i can easily recreate the problem locally using kubetest + local-up- running against containerd.
There are some notes on differences between how dockershim and other CRI impls handle this situation: have you run into this before?
Came up here:
I think what I said in that PR still applies.. I would expect kubectl -it to have a -d detach mode if background operation is the desired mode see docker exec -d option, and clarify what behavior is required of CRI exec sync.. up to 10sec pause for logging return value then return even if the in container exec process is still running? Note the hard coded start ops setting detach to false in docker shim: So it's using synch.. not detached.. but then later docker shim does this: before returning from the ExecSyncInContainer() request.
This seems to be a known issue. If an exec: 1) Spawns a long running background process; 2) Leaves the IO of the background process open after exits; 3) Containerd will wait until that IO to be closed before finishing the exec. It is arguable a bug in containerd, and the behavior of background process spawned by exec was never clearly defined. We've seen several production issues caused by this. Maybe we should: 1) Clearly define and document the behavior, and fix the test; 2) Pass timeout for probe (not related to this issue, but to protect the system). timeout is supported by many runtimes, but kubelet doesn't pass a proper value right now.
Among the callers of HandlerRunner#Run is the following from kubeGenericRuntimeManager#startContainer : We should find a proper value for the timeout.
We have a similar issue with CRI-O. I agree with that we should define this clearly and meanwhile setting a timeout makes sense.
I have opened to fix the tests.


Question: <del> // +build linux /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cm import ( "fmt" "strings" "time" "github.com/golang/glog" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/types" utilfeature "k8s.io/apiserver/pkg/util/feature" kubefeatures "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/events" evictionapi "k8s.io/kubernetes/pkg/kubelet/eviction/api" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" ) const ( defaultNodeAllocatableCgroupName = "kubepods" ) //createNodeAllocatableCgroups creates Node Allocatable Cgroup when CgroupsPerQOS flag is specified as true func (cm *containerManagerImpl) createNodeAllocatableCgroups() error { cgroupConfig := &CgroupConfig{ Name: cm.cgroupRoot, // The default limits for cpu shares can be very low which can lead to CPU starvation for pods. ResourceParameters: getCgroupConfig(cm.capacity), } if cm.cgroupManager.Exists(cgroupConfig.Name) { return nil } if err := cm.cgroupManager.Create(cgroupConfig); err != nil { glog.Errorf("Failed to create %q cgroup", cm.cgroupRoot) return err } return nil } // enforceNodeAllocatableCgroups enforce Node Allocatable Cgroup settings. func (cm *containerManagerImpl) enforceNodeAllocatableCgroups() error { nc := cm.NodeConfig.NodeAllocatableConfig // We need to update limits on node allocatable cgroup no matter what because // default cpu shares on cgroups are low and can cause cpu starvation. nodeAllocatable := cm.capacity // Use Node Allocatable limits instead of capacity if the user requested enforcing node allocatable. if cm.CgroupsPerQOS && nc.EnforceNodeAllocatable.Has(kubetypes.NodeAllocatableEnforcementKey) { nodeAllocatable = cm.getNodeAllocatableAbsolute() } glog.V(4).Infof("Attempting to enforce Node Allocatable with config: %+v", nc) cgroupConfig := &CgroupConfig{ Name:               cm.cgroupRoot, ResourceParameters: getCgroupConfig(nodeAllocatable), } // Using ObjectReference for events as the node maybe not cached; refer to #42701 for detail. nodeRef := &v1.ObjectReference{ Kind:      "Node", Name:      cm.nodeInfo.Name, UID:       types.UID(cm.nodeInfo.Name), Namespace: "", } // If Node Allocatable is enforced on a node that has not been drained or is updated on an existing node to a lower value, // existing memory usage across pods might be higher than current Node Allocatable Memory Limits. // Pod Evictions are expected to bring down memory usage to below Node Allocatable limits. // Until evictions happen retry cgroup updates. // Update limits on non root cgroup-root to be safe since the default limits for CPU can be too low. // Check if cgroupRoot is set to a non-empty value (empty would be the root container) if len(cm.cgroupRoot) > 0 { go func() { for { err := cm.cgroupManager.Update(cgroupConfig) if err == nil { cm.recorder.Event(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated Node Allocatable limit across pods") return } message := fmt.Sprintf("Failed to update Node Allocatable Limits %q: %v", cm.cgroupRoot, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) time.Sleep(time.Minute) } }() } // Now apply kube reserved and system reserved limits if required. if nc.EnforceNodeAllocatable.Has(kubetypes.SystemReservedEnforcementKey) { glog.V(2).Infof("Enforcing System reserved on cgroup %q with limits: %+v", nc.SystemReservedCgroupName, nc.SystemReserved) if err := enforceExistingCgroup(cm.cgroupManager, ParseCgroupfsToCgroupName(nc.SystemReservedCgroupName), nc.SystemReserved); err != nil { message := fmt.Sprintf("Failed to enforce System Reserved Cgroup Limits on %q: %v", nc.SystemReservedCgroupName, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) return fmt.Errorf(message) } cm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated limits on system reserved cgroup %v", nc.SystemReservedCgroupName) } if nc.EnforceNodeAllocatable.Has(kubetypes.KubeReservedEnforcementKey) { glog.V(2).Infof("Enforcing kube reserved on cgroup %q with limits: %+v", nc.KubeReservedCgroupName, nc.KubeReserved) if err := enforceExistingCgroup(cm.cgroupManager, ParseCgroupfsToCgroupName(nc.KubeReservedCgroupName), nc.KubeReserved); err != nil { message := fmt.Sprintf("Failed to enforce Kube Reserved Cgroup Limits on %q: %v", nc.KubeReservedCgroupName, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) return fmt.Errorf(message) } cm.recorder.Eventf(nodeRef, v1.EventTypeNormal, events.SuccessfulNodeAllocatableEnforcement, "Updated limits on kube reserved cgroup %v", nc.KubeReservedCgroupName) } return nil } // enforceExistingCgroup updates the limits `rl` on existing cgroup `cName` using `cgroupManager` interface. func enforceExistingCgroup(cgroupManager CgroupManager, cName CgroupName, rl v1.ResourceList) error { cgroupConfig := &CgroupConfig{ Name:               cName, ResourceParameters: getCgroupConfig(rl), } if cgroupConfig.ResourceParameters == nil { return fmt.Errorf("%q cgroup is not config properly", cgroupConfig.Name) } glog.V(4).Infof("Enforcing limits on cgroup %q with %d cpu shares and %d bytes of memory", cName, cgroupConfig.ResourceParameters.CpuShares, cgroupConfig.ResourceParameters.Memory) if !cgroupManager.Exists(cgroupConfig.Name) { return fmt.Errorf("%q cgroup does not exist", cgroupConfig.Name) } if err := cgroupManager.Update(cgroupConfig); err != nil { return err } return nil } // getCgroupConfig returns a ResourceConfig object that can be used to create or update cgroups via CgroupManager interface. func getCgroupConfig(rl v1.ResourceList) *ResourceConfig { // TODO(vishh): Set CPU Quota if necessary. if rl == nil { return nil } var rc ResourceConfig if q, exists := rl[v1.ResourceMemory]; exists { // Memory is defined in bytes. val := q.Value() rc.Memory = &val } if q, exists := rl[v1.ResourceCPU]; exists { // CPU is defined in milli-cores. val := MilliCPUToShares(q.MilliValue()) rc.CpuShares = &val } if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.HugePages) { rc.HugePageLimit = HugePageLimits(rl) } return &rc } // getNodeAllocatableAbsolute returns the absolute value of Node Allocatable which is primarily useful for enforcement. // Note that not all resources that are available on the node are included in the returned list of resources. // Returns a ResourceList. func (cm *containerManagerImpl) getNodeAllocatableAbsolute() v1.ResourceList { result := make(v1.ResourceList) for k, v := range cm.capacity { value := *(v.Copy()) if cm.NodeConfig.SystemReserved != nil { value.Sub(cm.NodeConfig.SystemReserved[k]) } if cm.NodeConfig.KubeReserved != nil { value.Sub(cm.NodeConfig.KubeReserved[k]) } if value.Sign() < 0 { // Negative Allocatable resources don't make sense. value.Set(0) } result[k] = value } return result } // GetNodeAllocatableReservation returns amount of compute or storage resource that have to be reserved on this node from scheduling. func (cm *containerManagerImpl) GetNodeAllocatableReservation() v1.ResourceList { evictionReservation := hardEvictionReservation(cm.HardEvictionThresholds, cm.capacity) result := make(v1.ResourceList) for k := range cm.capacity { value := resource.NewQuantity(0, resource.DecimalSI) if cm.NodeConfig.SystemReserved != nil { value.Add(cm.NodeConfig.SystemReserved[k]) } if cm.NodeConfig.KubeReserved != nil { value.Add(cm.NodeConfig.KubeReserved[k]) } if evictionReservation != nil { value.Add(evictionReservation[k]) } if !value.IsZero() { result[k] = *value } } return result } // hardEvictionReservation returns a resourcelist that includes reservation of resources based on hard eviction thresholds. func hardEvictionReservation(thresholds []evictionapi.Threshold, capacity v1.ResourceList) v1.ResourceList { if len(thresholds) == 0 { return nil } ret := v1.ResourceList{} for _, threshold := range thresholds { if threshold.Operator != evictionapi.OpLessThan { continue } switch threshold.Signal { case evictionapi.SignalMemoryAvailable: memoryCapacity := capacity[v1.ResourceMemory] value := evictionapi.GetThresholdQuantity(threshold.Value, &memoryCapacity) ret[v1.ResourceMemory] = *value case evictionapi.SignalNodeFsAvailable: storageCapacity := capacity[v1.ResourceEphemeralStorage] value := evictionapi.GetThresholdQuantity(threshold.Value, &storageCapacity) ret[v1.ResourceEphemeralStorage] = *value } } return ret } // validateNodeAllocatable ensures that the user specified Node Allocatable Configuration doesn't reserve more than the node capacity. // Returns error if the configuration is invalid, nil otherwise. func (cm *containerManagerImpl) validateNodeAllocatable() error { var errors []string nar := cm.GetNodeAllocatableReservation() for k, v := range nar { value := cm.capacity[k].DeepCopy() value.Sub(v) if value.Sign() < 0 { errors = append(errors, fmt.Sprintf("Resource %q has an allocatable of %v, capacity of %v", k, v, value)) } } if len(errors) > 0 { return fmt.Errorf("Invalid Node Allocatable configuration. %s", strings.Join(errors, " ")) } return nil } </del>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After running some tests, we're seeing that the combined usage of kubelet + docker can be around 2Gi. We need to be able to account for this in scheduling to avoid crashing these services. The first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the Windows nodes with too many containers. What you expected to happen: It should reduce node allocatable memory by kube-reserved + system-reserved. How to reproduce it (as minimally and precisely as possible): This logs an error in the kubelet log such as: code: Anything else we need to know?: /milestone v1.13 /assign feiskyer /sig windows This blocks a PR where I want to set it by default to 2Gi Environment: Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine
You must be a member of the github team to set the milestone. <detailsIn response to : Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: This logs an error in the kubelet log such as: code: /milestone v1.13 /assign feiskyer Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.13
/priority important-soon


Question: // either 0 or between 10000 and 2e9. // Please refer, https://github.com/kubernetes/kubernetes/pull/95345#discussion_r501630942 // for more information. <del> "UsageNanoCores":       gomega.SatisfyAny(gomega.BeZero(), bounded(10000, 2e9)), </del> <ins> "UsageNanoCores":       gomega.SatisfyAny(gstruct.PointTo(gomega.BeZero()), bounded(10000, 2e9)), </ins> "UsageCoreNanoSeconds": bounded(10000000, 1e15), }), "Memory": ptrMatchAllFields(gstruct.Fields{

links to specific failures in spyglass: /sig node
/triage accepted
/assign
any updates on this?
Will take a look at this week.
It looks has the same symptoms as - FYI Looks as cadvisor stats provider(e2e node still runs with docker) does not provide relevant stats. It looks like some cadvisor bug, because we only fethc the data from the cadvisor.
/area kubelet cadvisor
/priority important-soon
/cc
/milestone v1.22
/unassign /assign
Are there examples of recent flakes for this? Looking at it seems pretty green for this test over the past 2 weeks.
It does not happen often, but still happen, see -
Also, take into account that we mitigated one of the similar errors via
That's failing often enough to be considered flaky; I have seen this test fail frequently on presubmit jobs.
Flaking very frequently on the swap ubuntu periodic:
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
Taking a closer look. Some failures from Failure 1: Failure 2: Failure 3:
hmm, I'm not sure if I understand the failure correctly. It looks like the test gets back for and expects that it will match either zero or be = . The test code expects either 0 or between and 2e9. Based on error message, it in fact look like it's getting back 0: so I'm a bit confused why this check is failing. Am I misunderstanding the failure or does the line not do what we are expecting? /cc who looked who spent some time looking at this as part of
Also by the way, the failures in ubuntu swap job seem to be caused by a different reason than the kubelet master conformance job. Those failures seem to be due to MemoryStats: e.g. Perhaps memory stats may be impacted by swap being enabled?
I tried to repro the failures in by making the below patch (to force to always be zero). With this patch, the test fails consistently: despite the fact that zero should be valid and passing for the test So this makes me think that the problem is that the is not actually doing what we expect.
I think the problem is that is doing a uint64 pointer comparison to uint64.

expected is matcher expects The following patch to test results in test passing when zero is returned:
/reopen This appears to be flaking a lot per triage, I'm also seeing it on PRs
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since this is against milestone v1.22, let me close this and potentially open a new issue as the current failures don't match. /close
Closing this issue. <detailsIn response to : Since this is against milestone v1.22, let me close this and potentially open a new issue as the current failures don't match. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } case <-time.After(kl.resyncInterval): glog.V(4).Infof("Periodic sync") <del> if kl.pods == nil { continue } </del> } err := handler.SyncPods(kl.pods)

To reproduce: Start a cluster via hack/local-up- Start some pods Kill the cluster "docker ps -a" should show the pod containers running still Start the cluster again "etcdctl ls /registry" should show that there is no boundpods directory -- only minions and services "docker ps -a" should show the old pod containers still running Start a new pod After, say, 10s, "docker ps -a" should show the old containers dead /cc
I suspect the cause of this might be : which guarantees that the pod sync logic is never called if is nil. Need to get a repro to confirm.


Question: func isMountPointMatch(mp MountPoint, dir string) bool { return mp.Path == dir } <ins> // PathExists returns true if the specified path exists. // TODO: clean this up to use pkg/util/file/FileExists func PathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if os.IsNotExist(err) { return false, nil } else if IsCorruptedMnt(err) { return true, err } return false, err } </ins>

What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?

Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,


Question: SecurityContext: &v1.SecurityContext{}, }, }, <ins> NodeName: nodeName, </ins> }, }, },

Which jobs are flaking: pull-kubernetes-e2e-kind Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig apps
Failure at
Hi I looked into the failures and think it was because the test required the job succeeds 3 times with pseudorandom 50% failure chance within 15 minutes, while there is an exponential back-off delay (10s, 20s, 40s …) capped at 6 minutes before recreating failed pods. As 7 consecutive failures (1/128 chance) could exceed the timeout, the test failed intermittently because of "timed out waiting for the condition". For example, in the failure of , the job pods succeeded and failed in the following orders: After the last failure at , the delay has increased to 640s so no pod was created any more before the test failed at . I think it can be fixed by using a deterministic approach to make the job pods succeed and fail alternately, and its executing time can be reduced to tens of seconds stably. Assigning myself. /assign


Question: // UpdateSnapshot takes a snapshot of cached NodeInfo map. This is called at // beginning of every scheduling cycle. <ins> // The snapshot only includes Nodes that are not deleted at the time this function is called. // nodeinfo.Node() is guaranteed to be not nil for all the nodes in the snapshot. </ins> // This function tracks generation number of NodeInfo and updates only the // entries of an existing snapshot that have changed after the snapshot was taken. func (cache *schedulerCache) UpdateSnapshot(nodeSnapshot *Snapshot) error {

/sig scheduling What happened: reverted an old behavior to ensure Pod information is not lost when Nodes are recreated without its Pods getting recreated. A ghost Pod is kep in The problem is that we rely in this map for determining deleted Nodes when updating the snapshot in . The good thing is that the nodeinfo's are cloned into the snapshot, so we shouldn't have nil pointer accesses. But we might lose a few cycles trying to schedule on nodes that are deleted How to reproduce it (as minimally and precisely as possible): a node with a few pods. node without deleting pods. Result: The snapshot still has the node. Environment: Kubernetes version (use ): 1.18+
/assign
It actually produces the error: which would cause failed scheduling cycles (the map is not recreated, only the lists).


Question: OPENSTACK_IMAGE_NAME=${OPENSTACK_IMAGE_NAME:-CentOS7} # Downloaded image name for Openstack project <del> IMAGE_FILE=${IMAGE_FILE:-CentOS-7-x86_64-GenericCloud-1510.qcow2} </del> <ins> IMAGE_FILE=${IMAGE_FILE:-CentOS-7-x86_64-GenericCloud-1604.qcow2} </ins> # Absolute path where image file is stored. IMAGE_PATH=${IMAGE_PATH:-~/Downloads/openstack}

ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew kubetest.SkewTest 2022-04-14 15:13:42 +0000 UTC The error message is It's complaining about "file contents" and not "url contents". And because this is a skew test I am assuming this is due to The fix might be to cherry pick this to 1.23. No response /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've opened the cherry pick at if we decide that's the right route. The other options I can think of are to revert this change on master or to add the old environment variable back in as a file path. /assign /priority critical-urgent
Tracked the root of this to This should start passing once that merges. The cherry pick will also fix. We really need to get skew tests running on PRs.


Question: # Super annoying, the salt version on GCE is old enough that 'salt.cmd.run' # isn't supported {% if grains.cloud is defined and grains.cloud == 'aws' %} <del> {% set is_helium = salt.cmd.run('salt --version | grep -c Helium') %} </del> <ins> # Salt has terrible problems with systemd on AWS too {% set is_helium = '0' %} </ins> {% endif %} # Salt Helium doesn't support systemd modules for service running {% if pillar.get('is_systemd') and is_helium == '0' %}

On AWS, the kube-node-unpacker service is not starting, and so the kube-proxy is not starting, because the image is not registered. starts it. This is in : I suspect we're relying on salt to start a systemd service instead of using the workaround ;-) cc because will likely case AWS e2e to fail right now
I suspect this is actually the same as ; I suspect that the test for salt helium is failing and we are falling back to init.d
Obsoleted by

Not sure if this is causing a problem, but it looks like the version test for salt can't find salt: cc in case you're still having trouble with AWS e2e


Question: return false, nil } <del> if wantPrefixForEncryptAll != "" { deploymentName := fmt.Sprintf("deployment-%d", idx) _, err := test.createDeployment(deploymentName, "default") if err != nil { t.Fatalf("Failed to create test secret, error: %v", err) } rawEnvelope, err := test.readRawRecordFromETCD(test.getETCDPathForResource(test.storageConfig.Prefix, "", "deployments", deploymentName, "default")) if err != nil { t.Fatalf("failed to read %s from etcd: %v", test.getETCDPathForResource(test.storageConfig.Prefix, "", "deployments", deploymentName, "default"), err) } // check prefix if !bytes.HasPrefix(rawEnvelope.Kvs[0].Value, []byte(wantPrefixForEncryptAll)) { idx++ swapErr = fmt.Errorf("expected deployment to be prefixed with %s, but got %s", wantPrefixForEncryptAll, rawEnvelope.Kvs[0].Value) // return nil error to continue polling till timeout return false, nil } } </del> return true, nil }) if pollErr == wait.ErrWaitTimeout {

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/spf13/cobra" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: } if nc.EnforceNodeAllocatable.Has(kubetypes.KubeReservedEnforcementKey) { klog.V(2).Infof("Enforcing kube reserved on cgroup %q with limits: %+v", nc.KubeReservedCgroupName, nc.KubeReserved) <del> if err := enforceExistingCgroup(cm.cgroupManager, ParseCgroupfsToCgroupName(nc.KubeReservedCgroupName), nc.KubeReserved); err != nil { </del> <ins> if err := enforceExistingCgroup(cm.cgroupManager, cm.cgroupManager.CgroupName(nc.KubeReservedCgroupName), nc.KubeReserved); err != nil { </ins> message := fmt.Sprintf("Failed to enforce Kube Reserved Cgroup Limits on %q: %v", nc.KubeReservedCgroupName, err) cm.recorder.Event(nodeRef, v1.EventTypeWarning, events.FailedNodeAllocatableEnforcement, message) return fmt.Errorf(message)

start kubelet with below config: with below config: check the cgroup, get the following output: then, start kubelet.service, run as follows: why? Environment: Kubernetes version (use ): v1.14.1 Cloud provider or hardware configuration: OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g. ): SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/area kubelet /sig node
From cgroupManagerImpl#Exists : Can you turn on log V4 so that we can see the above log ? thanks
It seems if we change return value type to error, the caller would be able to incorporate error message from Exists func.
Thanks for sharing :) Hope we can help you debug! Is the information here
thanks for your rely. when I turn on log V4, logs as followed: Cgroup path is not expected: , so when I modify , it‘s ok. so Is this normal?
Ah, it looks like its appending automatically to the cgroup name. At the very least, I think the documentation could be clearer to indicate this will happen. An additional option could be adding the suffix only if doesn't already exist.
thank you
no problem :) If this issue is resolved, would you mind closing it? Thanks!
It might be worth considering adding a little code somewhere that simply strips off the ".slice" from the cgroup name supplied in configuration option and just normalizing the input in the configuration package itself (possibly with a warning that prints out stating cgroup should not end in ".slice")
I think that's a great idea. I will work on that, unless you want to :) /assign
thank you


Question: ginkgo.It("should function for node-Service: http", func() { config := e2enetwork.NewNetworkingTestConfig(f, true, false) ginkgo.By(fmt.Sprintf("dialing(http) %v (node) --> %v:%v (config.clusterIP)", config.NodeIP, config.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromNode("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromNode("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(http) %v (node) --> %v:%v (nodeIP)", config.NodeIP, config.NodeIP, config.NodeHTTPPort)) <del> config.DialFromNode("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromNode("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should function for node-Service: udp", func() { config := e2enetwork.NewNetworkingTestConfig(f, true, false) ginkgo.By(fmt.Sprintf("dialing(udp) %v (node) --> %v:%v (config.clusterIP)", config.NodeIP, config.ClusterIP, e2enetwork.ClusterUDPPort)) <del> config.DialFromNode("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromNode("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(udp) %v (node) --> %v:%v (nodeIP)", config.NodeIP, config.NodeIP, config.NodeUDPPort)) <del> config.DialFromNode("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromNode("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should function for endpoint-Service: http", func() {

What happened: final passing output: What you expected to happen: Probably if this test passes, it should be showing several real endpoints. I don't know exactly what it does, but im assuming that the output could be clarified a little. How to reproduce it (as minimally and precisely as possible): run and look at the logs. Anything else we need to know?: Environment: Kubernetes version (use ): 1.20 (sonobuoy)
/sig-network
/assign
/triage accepted
^ an interesting case here was that was but there was no SIG (typo in the command that did). should the label be only if there is label on the ticket? /sig network
the PR that adds triage/accepeted automation () was merged 3 hours ago, but it'll "actually" get rolled out in the next prow bump. Mostly when merges. Thanks for flagging though, appreciate it! :)
sorry, missed the actual question! :woman_facepalming: (need moar coffee!! :coffee: ) I don't think we should require a label for adding the label. In case someone comments with the following, we'd still want it to be valid.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
The above test cleans up the By directives and so on, After reading through the test more its clear that every test sets up a scaffold measures a property of that scaffold sometimes deletes something that is recovered in that scaffold (i.e. a pod that is behind a UDP service) So, the thing that was confusing I guess to me is that we validate empty conditions (i.e. no pods available) once we delete a nodePort svc. Meanwhile also noticed that the original work to clean up error reporting needed to cascade over to these tests as well.Have just finished all that, and also updated comments to reflect the Todo around the statistical computation (we have the ability to do extra polling via but we don't actually have any test that leverages this.As a follow on I guess it would be nice if we did have a test that confirmed that pods were load balanced evenly, esp with the new proliferation of different proxy technologies , I.e. a (or something similar


Question: // Add support for distributed tracing in the API Server APIServerTracing featuregate.Feature = "APIServerTracing" <ins> // owner: @linxiulei // beta: v1.30 // // Enables serving watch requests in separate goroutines. APIServingWithRoutine featuregate.Feature = "APIServingWithRoutine" </ins> // owner: @cici37 @jpbetz // kep: http://kep.k8s.io/3488 // alpha: v1.26

Optimize the memory usage of goroutine stack for serving watch requests. According to the of handling a watch request, the stack memory is 0x3768, which is very close to 16 KiB and Golang runtime will double it to 32 KiB once exceeded. In fact, it does exceed if we print the traceback deeper and take the fact that extra stack memory reserved by Golang itself into consideration. A naive optimization is to run as much as preparation handlers in a separate goroutine and then move the actual serving to another goroutine, so the former routine can free the stack memory earlier and make it more memory efficient. Given the extensive use of watch requests, 32 KiB per watch could add up to a very high amount of memory overhead for . A typical watch request commonly lasts for 30s or longer, which means this memory will be active for quite long. I see a great improvement in memory efficiency to do this optimization.
/cc
/sig api-machinery
/cc if you would like to take a look when you have time :) Thank you /triage accepted


Question: "k8s.io/utils/pointer" ) <del> // TODO(#105303): Add a test case to cover event registration for core API resources </del> <ins> // TestCoreResourceEnqueue verify Pods failed by in-tree default plugins can be // moved properly upon their registered events. func TestCoreResourceEnqueue(t *testing.T) { // Use zero backoff seconds to bypass backoffQ. // It's intended to not start the scheduler's queue, and hence to // not start any flushing logic. We will pop and schedule the Pods manually later. testCtx := testutils.InitTestSchedulerWithOptions( t, testutils.InitTestAPIServer(t, "core-res-enqueue", nil), nil, scheduler.WithPodInitialBackoffSeconds(0), scheduler.WithPodMaxBackoffSeconds(0), ) testutils.SyncInformerFactory(testCtx) defer testutils.CleanupTest(t, testCtx) cs, ns, ctx := testCtx.ClientSet, testCtx.NS.Name, testCtx.Ctx // Create one Node with a taint. node := st.MakeNode().Name("fake-node").Capacity(map[v1.ResourceName]string{v1.ResourceCPU: "2"}).Obj() node.Spec.Taints = []v1.Taint{{Key: "foo", Effect: v1.TaintEffectNoSchedule}} if _, err := cs.CoreV1().Nodes().Create(ctx, node, metav1.CreateOptions{}); err != nil { t.Fatalf("Failed to create Node %q: %v", node.Name, err) } // Create two Pods that are both unschedulable. // - Pod1 is a best-effort Pod, but doesn't have the required toleration. // - Pod2 requests a large amount of CPU resource that the node cannot fit. //   Note: Pod2 will fail the tainttoleration plugin b/c that's ordered prior to noderesources. pod1 := st.MakePod().Namespace(ns).Name("pod1").Container("image").Obj() pod2 := st.MakePod().Namespace(ns).Name("pod2").Req(map[v1.ResourceName]string{v1.ResourceCPU: "4"}).Obj() for _, pod := range []*v1.Pod{pod1, pod2} { if _, err := cs.CoreV1().Pods(ns).Create(ctx, pod, metav1.CreateOptions{}); err != nil { t.Fatalf("Failed to create Pod %q: %v", pod.Name, err) } } // Wait for the two pods to be present in the scheduling queue. if err := wait.Poll(time.Millisecond*200, wait.ForeverTestTimeout, func() (bool, error) { return len(testCtx.Scheduler.SchedulingQueue.PendingPods()) == 2, nil }); err != nil { t.Fatal(err) } // Pop the two pods out. They should be unschedulable. for i := 0; i < 2; i++ { podInfo := nextPodOrDie(t, testCtx) fwk, ok := testCtx.Scheduler.Profiles[podInfo.Pod.Spec.SchedulerName] if !ok { t.Fatalf("Cannot find the profile for Pod %v", podInfo.Pod.Name) } // Schedule the Pod manually. _, fitError := testCtx.Scheduler.Algorithm.Schedule(ctx, nil, fwk, framework.NewCycleState(), podInfo.Pod) if fitError == nil { t.Fatalf("Expect Pod %v to fail at scheduling.", podInfo.Pod.Name) } testCtx.Scheduler.Error(podInfo, fitError) // Scheduling cycle is incremented by one after NextPod() is called, so // pass a number larger than i to move Pod to unschedulableQ. testCtx.Scheduler.SchedulingQueue.AddUnschedulableIfNotPresent(podInfo, int64(i+10)) } // Trigger a NodeTaintChange event. // We expect this event to trigger moving the test Pod from unschedulableQ to activeQ. node.Spec.Taints = nil if _, err := cs.CoreV1().Nodes().Update(ctx, node, metav1.UpdateOptions{}); err != nil { t.Fatalf("Failed to remove taints off the node: %v", err) } // Now we should be able to pop the Pod from activeQ again. podInfo := nextPodOrDie(t, testCtx) if podInfo.Attempts != 2 { t.Fatalf("Expected the Pod to be attempted 2 times, but got %v", podInfo.Attempts) } if got := podInfo.Pod.Name; got != "pod1" { t.Fatalf("Exepcted pod1 to be popped, but got %v", got) } // Pod2 is not expected to be popped out. // Although the failure reason has been lifted, it still won't be moved to active due to // the node event's preCheckForNode(). podInfo = nextPod(t, testCtx) if podInfo != nil { t.Fatalf("Unexpected pod %v get popped out", podInfo.Pod.Name) } } </ins> var _ framework.FilterPlugin = &fakeCRPlugin{} var _ framework.EnqueueExtensions = &fakeCRPlugin{}

Currently, uses ServiceAffinity plugin to cover the event registration logic: But along with the removal of ServiceAffinity plugin, this test will also be removed . We will come up with a supplemental test to cover the event registration logic. /sig scheduling
/assign /triage accepted
/kind cleanup


Question: err := store.Delete(ctx, key, out, tt.precondition, storage.ValidateAllObjectFunc, nil) if tt.expectInvalidObjErr { if err == nil || !storage.IsInvalidObj(err) { <del> t.Errorf("%s: expecting invalid UID error, but get: %s", tt.name, err) </del> <ins> t.Errorf("expecting invalid UID error, but get: %s", err) </ins> } return } if err != nil { <del> t.Fatalf("%s: Delete failed: %v", tt.name, err) </del> <ins> t.Fatalf("Delete failed: %v", err) } // We expect the resource version of the returned object to be // updated compared to the last existing object. if storedObj.ResourceVersion == out.ResourceVersion { t.Errorf("expecting resource version to be updated, but get: %s", out.ResourceVersion) </ins> } <del> ExpectNoDiff(t, fmt.Sprintf("%s: incorrect pod", tt.name), storedObj, out) </del> <ins> out.ResourceVersion = storedObj.ResourceVersion ExpectNoDiff(t, "incorrect pod:", storedObj, out) </ins> key, storedObj = TestPropagateStore(ctx, t, store, &example.Pod{ObjectMeta: metav1.ObjectMeta{Name: "foo", UID: "A"}}) }) }

While working on generalizing tests to somehow better define spec for storage interface (and share them across all implementation of storage.interface - in particular etcd3 and watchcache), I found an unexpected inconsistency in Delete operation in etcd3. What is happening is that it effectively returns the state of the object from just before the deletion: However, this is inconsistent with how our watch is working, which is returning the state of the object from just before the deletion, but it sets the RV to the moment of deletion (which Delete call isn't doing). This sounds like a bug to me, but given this is a long-standing behavior we may need to take additional care when fixing this. /kind bug /sig api-machinery /priority important-soon /cc
I opened as a fix for it.
/triage accepted


Question: "container_manager_unsupported.go", "container_manager_windows.go", "fake_internal_container_lifecycle.go", <ins> "helpers.go", </ins> "helpers_linux.go", "helpers_unsupported.go", "internal_container_lifecycle.go", <del> "node_container_manager.go", </del> <ins> "node_container_manager_linux.go", </ins> "pod_container_manager_linux.go", "pod_container_manager_stub.go", "pod_container_manager_unsupported.go",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After running some tests, we're seeing that the combined usage of kubelet + docker can be around 2Gi. We need to be able to account for this in scheduling to avoid crashing these services. The first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the Windows nodes with too many containers. What you expected to happen: It should reduce node allocatable memory by kube-reserved + system-reserved. How to reproduce it (as minimally and precisely as possible): This logs an error in the kubelet log such as: code: Anything else we need to know?: /milestone v1.13 /assign feiskyer /sig windows This blocks a PR where I want to set it by default to 2Gi Environment: Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine
You must be a member of the github team to set the milestone. <detailsIn response to : Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: This logs an error in the kubelet log such as: code: /milestone v1.13 /assign feiskyer Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.13
/priority important-soon


Question: // or a file device.  Block devices can be used directly, but file devices must // be made accessible as a block device before using. func (u *storageosUtil) AttachVolume(b *storageosMounter) (string, error) { <ins> klog.V(4).Infof("attaching StorageOS volume %q with namespace %q", b.volName, b.volNamespace) </ins> if err := u.NewAPI(b.apiCfg); err != nil { return "", err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When a PVC is created and the PV mounted into a Pod using the StorageOS in-tree volume driver, the StorageOS CLI & UI do not report that the volume is mounted. The PV is created and mounted into the Pod successfully, and all k8s functionality works as expected. However, since the correct mount status is not visible to administrators using the StorageOS UI or CLI, they may make changes to or delete a mounted volume without realising, and the safeguards (must use to delete a mounted volume) do not trigger. This is a regression caused by , released in 1.14.0. What you expected to happen: The StorageOS CLI should report volume mount information (node name, mount timestamp). How to reproduce it (as minimally and precisely as possible): StorageOS node containers with CSI disabled: application Pod with a PVC using the StorageOS StorageClass: StorageOS to list volumes: Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.0 and later Cloud provider or hardware configuration: any OS (e.g: ): any linux (incl. Ubuntu 18.04) Kernel (e.g. ): any linux (incl. 4.15.0-33-generic) Install tools: any (incl. minikube) Network plugin and version (if this is a network-related bug): N/A Others: Requires StorageOS deployment /sig storage /assign


Question: interference, deadlines, and so on. Workload-specific requirements will be exposed through the API as necessary.`, Run: func(cmd *cobra.Command, args []string) { <del> if err := runCommand(cmd, args, opts); err != nil { </del> <ins> if err := runCommand(cmd, args, opts, registryOptions...); err != nil { </ins> fmt.Fprintf(os.Stderr, "%vn", err) os.Exit(1) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : As the is now implementable, it will be easier to add out of tree scheduler. And we are trying to use it to implement our plugins. The issue tracks following problems: Implement interface same as Custom Scheduler Plugins (out of tree)  to support out of tree scheduler Add support for scheduler to load and register user’s out of tree scheduling plugins that could be configured by admins. We could use to implement it, however it has some drawbacks as pointed out, e.g. We need find other ways, e.g. . /sig scheduling /assign
I did some investigation. It seems that is a better choice for the second part of this issue. I am making a prototype for it. Need I send a proposal at first? /cc
Just wrote a small proposal for it


Question: pod := getHugepagesTestPod(f, limits, mounts, volumes) <del> ginkgo.By("by running a guarantee pod that requests hugepages") </del> <ins> ginkgo.By("by running a test pod that requests hugepages") </ins> testpod = f.PodClient().CreateSync(pod) }) // we should use JustAfterEach because framework will teardown the client under the AfterEach method ginkgo.JustAfterEach(func() { <ins> ginkgo.By(fmt.Sprintf("deleting test pod %s", testpod.Name)) f.PodClient().DeleteSync(testpod.Name, metav1.DeleteOptions{}, 2*time.Minute) </ins> releaseHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The pod passed the scheduler(moved from the Pending phase) and failed to start because of the kubelet with the error: It means that the scheduler and kubelet get a different view of requested resources. I expect that if the scheduler passed the pod and the pod was started it should not fail under the kubelet because of insufficient resources. It failed under the CI - Steps: configure hugepages under the node(5 - 2Mi-hugepages) start the pod, that requests 6Mi of 2Mi-hugepages delete the namespace where the pod started restart the kubelet start an additional pod, that requests 6Mi of 2Mi-hugepages It happens very rarely, something around 10%. The problem that the scheduler and kubelet are using different mechanisms and structs to store active pods requests. Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am very interested in this question, I will try to analyze it.
I think some delay between the time when kubelet discovers the resource and the time when resource value is updated on the API server is expected. We can't guarantee that kubelet sends updated resource to the API server, API server receives the update and updates resource value in no time, can we?
the scheduler should see the same picture as the kubelet, so why does it pass the scheduler and failed under the kubelet? Do you know why do we need the same checks under two different places(kubelet and scheduler), why it not enough to check available resources only under the scheduler?

/sig node scheduling
are there any static pods being created in that test? static pods may cause this.
Nope, regular pods with the
/cc
I can not see the issue happens under the 1.21, I will re-open the issue if will encounter it again.
I can still see it in the recent logs. Here is an example from
Feel free to re-open it. I just checked that the hugepages job is much more stable than it was under 1.20.
I can't reopen it it seems. Can you?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue became a showstopper for promoting . Can you help to fix it?
I've noticed a lot of similar error messages in the kubelet log of successfull tests. Here is an example: taken from the latest available test run: The fact that node capacity for the requested hugepages is 0 for 3 pods in a row makes me suspect that the test creates pods requesting hugepages that are not allocated or something similar. Can you help to investigate this?


Question: - name: SchedulingRequiredPodAffinityWithNSSelector defaultPodTemplatePath: config/pod-affinity-ns-selector.yaml <del> featureGates: PodAffinityNamespaceSelector: true </del> workloadTemplate: - opcode: createNodes countParam: $initNodes

/sig scheduling in the job is failing. success run Due to this failure, we cannot see any scheduler metrics results from the perf-dash. master
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind bug /kind flake /kind failing-test
/assign /remove-kind flake
will fixed by


Question: var ( dInfo = driver.GetDriverInfo() cs    clientset.Interface <ins> l     *stressTest </ins> ) <ins> // Check preconditions before setting up namespace via framework below. </ins> ginkgo.BeforeEach(func() { <del> // Check preconditions. </del> if dInfo.StressTestOptions == nil { e2eskipper.Skipf("Driver %s doesn't specify stress test options -- skipping", dInfo.Name) }

/sig storage What happened: When you interrupt the e2e storage stress test, namespaced objects are cleaned up, but non-namespaced objects like StorageClasses are not cleaned up. What you expected to happen: Non-namespaced objects like StorageClasses are cleaned up so long as the user doesn't interrupt a second time. How to reproduce it (as minimally and precisely as possible):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, // PV agnostic path negative test cases { <del> testName:        "fail: missing json, fail to retrieve driver and volumeID from globalpath", volID:           "project/zone/test-vol1", </del> <ins> testName:        "success: json file doesn't exist, unmount device is skipped", </ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", jsonFile:        "", stageUnstageSet: true, <del> shouldFail:      true, </del> <ins> createPV:        true, </ins> }, { testName:        "fail: invalid json, fail to retrieve driver and volumeID from globalpath",

Hello, I've recently been testing the reliability of storage for k8s v1.25.0. A strange phenomenon occurs occasionally during the test: the volume cannot be detached after the PVC and pod are deleted(the reclaimPolicy of pv is delete. So I first went to check the corresponding and objects and found that they both existed. Then I checked the kube-controllerManager and kubelet logs and found that kubelet logs kept printing: But I found that at the beginning, the volume was unmountDevice succeed, as shown in the following kubelet log: it is so wired ! Then I some log variables to the unmountDetachDevices method: The results show that And I've drawn up a time-series chart of the contents of the log. ! Let's go back to the unmountDetachDevices method. According to the above-mentioned log content, the first operation successfully changes of the corresponding attachedVolume to , but the function in the subsequent operation still uses stale DeviceMountState. that the unmountDevice operation is not performed after the first unmountDevice operation succeeds. multiple unmountDevice operations are allowed, ensure that the correct pvname can be obtained in the method. the volume is detached after the pod and PVC objects are deleted This problem can be reproduced by repeatedly deleting and recreating pods that require PVC in k8s 1.25.0 cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage
we can't safely assume that the json is always there.
ping
Thanks for the detailed analysis . I'm still sorting through this, do you think that is involved with this?
Yes. look at this log content: The global mount path is changed in the As a result, the method obtains incorrect pvName(e6104b10fea107e004e3307b0444fd7b775fedd804ab47aa997118b7927c32e0). The unmountDevice operation may be performed multiple times in [1.19 - 1.25] kubernetes versions. This problem does not occur before because can still obtain the correct pvName from the global mount path. However, this problem occurs after is merged. https://user- <br</brAfter the first unmountDevice operation is performed successfully, the file is deleted.

So the subsequent unmountDevice operation will perform method because file doesn't exist.
Awesome, thanks. Let me find someone to work on this.
I ll take a look at the issue, since its related to the change I made Note: this change needs the node to drained before upgrading to a version with this change . Please check the release notes
Thanks a lot !
yep, I always reproduce this problem in a newly created k8s 1.25.0 cluster.
Hihas there been any progress on this issue?
/priority important-soon
/triage accepted
Hihas there been any progress?
I also hit this error after pvc and pod deleted on k8s 1.25: and then these two volumes cannot be unmounted on the node forever: I tried to delete pv, it's stuck in terminating state forever
restart the kubelet on the node in problem could mitigate this issue, but we really need to fix this issue ASAP, I could repro this issue easily when running 1000 disk attach/detach test
I think this is again a TOCTOU bug in unmountDetachDevices.
The Unmount operation is , thus the state may not be set as before next reconcile, so it is possible that unmountDevice operation performed multiple times. So the root cause of this issue is that got wrong PV name, since changed global mount path.
I have found a similar issue in our environment, which is based on Kubernetes 1.25.3. If I install a Pod with a PVC, and I force move the Pod with NodeAffinity around the cluster, around 10-20 retries the Pod does not come up: the VolumeAttachment is bound to the old node and is not moved by the system anymore. If this is the same issue as reported here I do not know, but the logs are similar: From the Worker's kubelet where the volumeattachment is bound to: Also I see lots of these: But, if this is the same, I would like to know that the referenced PR was merged into exactly which Kubernetes version, as I have found in the RELEASE NOTES, that it was in 1.24.0-alpha: But we could not reproduce the issue on Kubernetes 1.24.2. How come?
As per the logs, it's the same issue. Kubelet operates in a reconcile loop, and is an async operation.

This bug only happened when is operated for multiple times, so it cannot be 100% reproduced. However, if the possibility of reproduction in 1.25 is much higher than the cluster lower than 1.25, we should figure out whether there are some other changes that makes this bug happened more frequently.
I have 1K disk attach/detach to 40 nodes test, if it runs multiple times, it could 100% repro, we could use this patch for testing, if 100 test runs pass, then it could prove the fix really works.
Sure. I mean not every single Mount/Unmount operation will lead to this error, but if we do it enough times, then it could be 100% to reproduce.
May I ask what could be a workaround in production systems or even CI systems on this issue? Our CI is struggling with this and other random issues, to the point we would like to stabilize it by kicking the system in a way to fix things... I was trying a kubelet restart on the Worker where the Mount is stuck... but many cases the kubelet restart itself hangs or does not produce a correct restart. So what could be a workaround if we find out that there is a hung volumeattachment?
restart kubelet, and if restart kubelet hangs, reboot the node.
Volumeattachment failed to detach from old node When statefulset application with pvc scheduled on worker1 during deployment its volumeattachment bounds to worker1 when same application upgraded scheduled on another worker node say worker2 and struck in ContainerCreating state and volumeattachment failed to detach from old node .

when pod described: Events: Warning FailedMount 23m (x820 over 6d2h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[ datadir]: timed out waiting for the condition Warning FailedMount 5m22s (x3069 over 6d3h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition Environment: NAME="SLES" VERSION="15-SP4" VERSIONID="15.4" PRETTYNAME="SUSE Linux Enterprise Server 15 SP4" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp4" DOCUMENTATIONURL=""' Kubernetes version: v1.25.3 kernel: 5.14.21-.24.41-default Storage: Rook: v1.9.12 ceph: 16.2.10 logs of kubelet: Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ]

"Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85-
FYI. this issue is fixed in 1.24.14, 1.25.10, 1.26.5, and has been fixed in v1.27.0
Hello , can you have a look at this reconciler run's operationExecutor fails to umount volume due to missing vol- It appears to be Kube v1.22 not .25 to boot is this the same issue?
I think they are the same issue and I have left some comments in
Issue reoccurs in kubelet version


Question: // On success, objPtr would be set to the object. func decode(codec runtime.Codec, versioner storage.Versioner, value []byte, objPtr runtime.Object, rev int64) error { if _, err := conversion.EnforcePtr(objPtr); err != nil { <del> panic("unable to convert output object to pointer") </del> <ins> return fmt.Errorf("unable to convert output object to pointer: %v", err) </ins> } _, _, err := codec.Decode(value, nil, objPtr) if err != nil { return err } // being unable to set the version does not prevent the object from being extracted <del> versioner.UpdateObject(objPtr, uint64(rev)) </del> <ins> if err := versioner.UpdateObject(objPtr, uint64(rev)); err != nil { klog.Errorf("failed to update object version: %v", err) } </ins> return nil }

Remove panic in , please return error instead following .
/sig api-machinery /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Please also add some log if returns error
/remove-kind bug /kind cleanup


Question: pvReadCmd  string = "cat " + pvTestFile ) <del> func (t *PersistentVolumeUpgradeTest) createGCEVolume() *v1.PersistentVolumeSource { diskName, err := framework.CreatePDWithRetry() framework.ExpectNoError(err) return &v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:   diskName, FSType:   "ext3", ReadOnly: false, }, } } </del> func (t *PersistentVolumeUpgradeTest) deleteGCEVolume(pvSource *v1.PersistentVolumeSource) error { return framework.DeletePDWithRetry(pvSource.GCEPersistentDisk.PDName) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --We should have an e2e to test issue . The test needs to delete a PVC while a pod is still using it, and then restart the controller manager. /kind bug /sig storage
CC
/assign unofficially so I can find this later.


Question: }, "message": { "type": "string", <del> "description": "String detailing the error encountered during Attach or Detach operation. This string maybe logged, so it should not contain sensitive information." </del> <ins> "description": "String detailing the error encountered during Attach or Detach operation. This string may be logged, so it should not contain sensitive information." </ins> } } },

This came up as a follow up to Also rather than opening 5 different issues - we are going to use this github issue to track all follow up items mentioned here -
/sig storage
/assign /assign
: PR
and the rest is fixed in


Question: *resourceVersion = rc.ResourceVersion // Sync even if this is a deletion event, to ensure that we leave // it in the desired state. <del> glog.V(4).Infof("About to sync from watch: %v", rc.Name) </del> <ins> glog.V(4).Infof("About to sync from watch: %q", rc.Name) </ins> if err := rm.syncHandler(*rc); err != nil { util.HandleError(fmt.Errorf("unexpected sync error: %v", err)) }

I've noticed a issue where if you have multiple replicated slave pods that don't have a HOST assigned and you reduce the number of replicas in the replicationController definition for those pods, sometimes the currently assigned (and running) pods get killed and the unassigned pod will get assigned to run in the killed pods place. The issue can be reproduced using the guestbook example provided in the repo. Steps to reproduce: (Note this is not always deterministic and may need multiple attempts) the master pod the master service the replicated slaves the Slave Controller definition to only have 1 replicas the Slave Controller with new definition The expected behavior should be that the scheduler is able to determine that unassigned pods are more viable candidates when reducing number of replicas. It seems to be more likely that since this issue is not deterministic, the ordering in which the command returns result is not always deterministically in the same order
: please can you look at this or suggested someone else? Thank you.
Unassign myself from this bug since I cannot get working on this soon. Also remove kubelet from the label since it is not kubelet issue. This is obviously a replicaController issue. cc/ Instead of destroying pod in pending state, replicaController unbound running pod from a node. Kubelet running on such node has this new desire state and kill unwanted pod to match it.
Yes, the replication controller should be smarter, taking into account Phase, Readiness, recency, ... This will be a significant problem for auto-scalers, but I expect manual scaling to be infrequent. I doubt we'll get to this soon. A PR would be welcome. Place to change is here:
Hi, I am final year Master's student of International Institute of Information Technology, Hyderabad. I am familiar with docker and containers. I have basic understanding of Go language with some hands on Docker API. This project seems interesting and a solution to this will also be a good addition to Kubernetes. So I would like to pick this as my GSoC project. I have started reading about Replication Controller and Pods and going though the code. I would appreciate if someone has any inputs for me on this. Thanks, Swapnil Patil