Question: } } } <del> if err := kl.setNodeStatus(node); err != nil { return nil, err } </del> <ins> kl.setNodeStatus(node) </ins> return node, nil }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.3.8 custom build - the code seems to still be in place in master. Environment: Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): VERSION="14.04.5 LTS, Trusty Tahr" Kernel (e.g. ): 4.4.0-38-generic ~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: Custom Others: What happened: When the OpenStack Nova API is not accessible Nodes are marked as "NotReady" The OpenStack cloud providers seems to call Nova on a schedule to get information. If the API is not contactable, kubelet fails to call back to the masters, and gets marked as NotReady. Masters also call the Nova API, and when it fails, it can mark all nodes as NotReady. What you expected to happen: Nothing - the cluster should still work, regardless of Nova's API How to reproduce it (as minimally and precisely as possible): Install Kubernetes using the OpenStack provider. On the OpenStack controller stop the nova-api process.


Question: err := cs.CoreV1().Services(ns).Delete(context.TODO(), serviceName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete service: %s in namespace: %s", serviceName, ns) }() <del> _, err := jig.CreateTCPServiceWithPort(nil, 80) </del> <ins> svc, err := jig.CreateTCPServiceWithPort(nil, 80) </ins> framework.ExpectNoError(err) validateEndpointsPortsOrFail(cs, ns, serviceName, portsByPodName{})

The test only validates that the API objects are created E2e tests must verify the behaviour, i.e. that the Service and Endpoints actually work sending traffic. The file is not well organized and there is a lot of duplicate code, one possibility is to use current Services helper has a function to test the reachability that iterates over all service ports However, something that needs to be addressed is that the pods that are backing the service doesn't have any process listening on those ports
/help wanted /sig network /priority important-soon /triage accepted
I think that there are more tests that doesn't verify the reachability , those should be modified too /help-wanted
/help-wanted
/assign
/triage accepted
/assign /unassign I will take on this since it was more difficult than I expected, and is a Conformance test.. I assigned you to , that is similar to this, but creating a new test from scratch


Question: name string args args cap  int <ins> len  int </ins> }{ { name: "empty",

has by default, and since 1.22 has the same. They seem to return wrong results in certain situations combined with / . And / also use , so they also don't work correctly. outputs partial list of pods on the node, doesn't evict some of the pods. To make it work correctly, I had to specify works correctly with any , since it's just a performance optimizing parameter. I can't provide an actual concrete example (no = default ), but instead I show how to reproduce it with a smaller I believe that the same problem occurs with create two deployments. there are 5 pods in each. with --selector=app=another-server and , it shows only 4 pods. I could confirm that the problem happens in both case of and regardless namespace, even with , or <details</details<detailsGCP / GKE Standard </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig node
I think this is indeed a bug, after some simple testing, it seems to be an bug unique to 1.24, let me fix it. /assign
/assign
I can reproduce this error in a 1.24 cluster, but not in a 1.23.6 cluster. All clusters are created via kind, and I'm guessing gke may have cherry-picked some commits or had some of their own changes to their branch. So I think this fix should at least cherry-pick to 1.23 to help other distributions fix this issue.
/remove-sig api-machinery


Question: github.com/golangplus/fmt => github.com/golangplus/fmt v0.0.0-20150411045040-2a5d6d7d2995 github.com/golangplus/testing => github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e github.com/google/btree => github.com/google/btree v1.0.0 <del> github.com/google/cadvisor => github.com/google/cadvisor v0.38.5 </del> <ins> github.com/google/cadvisor => github.com/google/cadvisor v0.38.6 </ins> github.com/google/go-cmp => github.com/google/go-cmp v0.5.2 github.com/google/gofuzz => github.com/google/gofuzz v1.1.0 github.com/google/martian => github.com/google/martian v2.1.0+incompatible

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I restart containerd.service, kubelet can not connect to it again and cause 100% cpu usage. there are many logs like this: I can show container info via crictl at the meantime and restart kubelet.service can resolve the problem. What you expected to happen: kubelet can reconnect to containerd after containerd restart. How to reproduce it (as minimally and precisely as possible): cpu usage and syslog. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: armbian(debian on arm64) OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Reproduce: can reproduce the issue by restarting when is the container runtime another setup where was used used as container runtime, didn't experience the issue by restarting , but restarted the underlying , same issue shown up again and had to restart to resolve the issue. Will continue with the investigation, might need to reconnect to after the socket is broken.
To me it seems there are a couple of problems here. 1. kubelet does not reconnect to the containerd socket. 2. The reconnect is agressive and consumes a lot if cpu. I don't know the cause for 1, but for 2 this might be an indication that the retry interval should be relaxed.
Spent a few hours on the investigation, it's likely to be some internal states being messed up in .
TL;DR: It should be related to gRPC.

Hacked the vendored gRPC mod and changed its to something like this: Then CPU only spike for a few seconds after restart then comes back to normal instead of failing the all the time. kubelet passes a customized dialer to the gRPC client. I logged the address during kubelet startup: As you can see, the address of the is at . Before restarts, I can see from log that every time, the gRPC function is using the customized dialer that kubelet passed: However, once got restarted, the address of the dialer is changed: And that function at keeps failing. As I forced to use the original customized dialer, the issue is gone. Without too much knowledge on gRPC, my blind guess is as is passed by reference to gRPC client, some component could modify it during the process lifetime and the change got persisted. Anyway, I'll check with gRPC team on this issue and see if there's already a fix on their side. If so then I can port the fix here.
/cc
Hi, are there any updates on this. I am experiencing what appears to be the same issue. If I restart containerd while the kubelet is running, my kubelet logs get flooded with: Restarting the kubelet appears to be the only way to re-establish the connection with containerd. Version information is:
Hi thanks for the info, I'm still working on the root cause. As I mentioned above, it's something related to gRPC. I collected the object info, this is the object before containerd restart: ! And this is the object after containerd restart: ! There's one thing made me concern is, the connection state is still 2 (i.e. , see ) after the failure. I'm not sure if this is related to the cause but anyway it still should be something related to gRPC. Btw I've followed their to use the latest gRPC client without customized dialer and still see this issue, will still work on the root cause.
We're seeing this now when we update our nodes to 1.19 where our Kata Containers install does a . Our current workaround is to also do , but would be good to get this fixed.
Is this MR relative to the problem?

No I don't think so, I've cherry-picked that change and tried in my environment weeks back, and the issue is still there. I only get around the issue by modifying the gRPC code as I mentioned in the above, I'll continue to check which component modified the function in after thanksgiving.
For the sake of priority. This hit us hard these days too, as security updates for container.d triggered a restart and a ton of nodes flooded the central log management. ;) Thanks for working hard on a fix.
Same here. Two systems were affected by automatic container.d upgrades & restarts. But now I wonder if - generally - "unattended-upgrades" (Ubuntu 18.04) is a wise thing to do? Or should the OS for kubernetes deployments always be updated manually & controlled? I can not find best practices for this in the documentation.
After the recent CVE-2020-15257many users will upgrade and restart containerd. It would be a big problem.
I see, will prioritize on the root cause and fix. At the mean time, you need to restart after you restart .
Had another round of debugging, and made one step towards the root cause. I following snipple to the as I mentioned in previous : When I reproduced the error, I got following information for the dialer function: That leads to cadvisor, which is the only component that uses the in the repo: So I'm convinced that this should be related to and will update the thread once I've made further progress.
EOD updates: issue is indeed caused by cadvisor, reproduced the issue in this . confusing timeout in log should be . root cause might be a bug for using context that already exceeded deadline in cadvisor. To validate that, I need to explore all methods of this : Will work on the fix on cadvisor for the rest of the week.
Fix was made in Copying from to ensure we can capture it on the issue here: I've cut new cAdvisor releases (v0.37.3 and v0.38.6) with this fix cherrypicked. We should get the cAdvisor fix back into kubernetes.

We'll need 3 PRs to k/k with following updates: [x] k/k master to update cAdvisor to v0.38.6 -- [x] k/k release-1.20 to update cAdvisor to v0.38.6 -- [x] k/k release-1.19 to update cAdvisor to v0.37.3 --
Thanks for fixing this one, definitely a tricky one to track down.
Thanks for all the patience and team efforts on getting this issue fixed, also credits to on the guides and discussions, won't be able to get it done without your helps!
The fix was applied on master as well as cherrypicked into and releases.
I remember you told me, but I forgot. Does issue exist in 1.18 and 1.17?
I don't believe 1.18 or 1.17 should be affected. I think this issue was introduced with which commit was included in cAdvisor v0.37 release. cAdvisor v0.37 was vendored into k8s 1.19.


Question: awscredentials "k8s.io/kubernetes/pkg/credentialprovider/aws" "k8s.io/kubernetes/pkg/types" "k8s.io/kubernetes/pkg/util/sets" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> "k8s.io/kubernetes/pkg/volume" )

In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt.

calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it.

PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.


Question: } func NewDockerCache(client DockerInterface) (DockerCache, error) { <del> containers, err := GetKubeletDockerContainers(client, false) if err != nil { return nil, err } </del> return &dockerCache{ client:        client, <del> cacheTime:     time.Now(), containers:    containers, </del> updatingCache: false, }, nil }

I launched a cluster that was unhealthy and was trying to diagnose why when I ran across an interesting kubelet error. On the master, The validate endpoint is showing that the single node in the cluster cannot be reached: From on the node I see: and the kubelet appears to keep running but never recovers.
/cc
If I manually kill the kubelet () and then run on the master (to restart it) then it runs without re-printing the above error and the master's validate endpoint shows that the node is healthy. It would be much nicer if the kubelet could recover on its own without needing me to manually intervene.
I encountered this a second time this evening but with a slightly different error in the file The symptoms and remediation were the same.
This was on the 0.12.0 release.
Ok, the initial error reported above is due to docker is not running. The issue here is not kubelet is not running, instead of docker is shutdown. Why docker is shutdown here? because we upgrade to 1.5 docker?
I lowered the priority to P1. We recently monit to babysitting both docker and kubelet. Monit will check both docker and kubelet if running every 5 mins (). In your failure case, docker should be restart by monit. We can discuss the interval here. Maybe reduce it to 1 min.
Reducing the monitoring period SGTM.
How long you have wait when you query /validate? If it takes too long for a cluster being initialized, that is a separate issue and we should fix.
Ok, introduced DockerCache in kubelet, which makes thing worse. Basically it constructs a DockerCache at kubelet initialization time. If docker is not running, kubelet bailed out too. I am assigning this to you. I think we shouldn't fail kubelet initialization here just because of docker is shutdown. Kubelet should report NodeStatus including docker's status.
I will address the monit' s period issue in
Trying to address your questions in order: We have intermitent issues with docker not starting. is working on tracking it down (see ). On these nodes docker was running as was the kubelet, but the kubelet wasn't reachable by the master because it wasn't listening on the expected port. What does monit check? That the process is alive?

Can it be configured to make sure it is "healthy" (for some definition of healthy)? If it just checks that the process is up, maybe kubelet should exit when it can't initialize properly and let monit restart it. We wait up to 10 minutes for /validate to report that all cluster components are healthy. The cluster is "initialized" in the sense that all components are running, but the cluster isn't functional because all components are not healthy. In this case manually bouncing the kubelet process pretty much immediately moved the node to healthy making the entire cluster functional. I did the manual checks 20-30 minutes after the initial 10 minute timeout, meaning that even at a 5 minute interval monit had plenty of chances to rectify the broken kubelet (and appeared to not do so).
Another data point: I just looked at a cluster that was launched at 8:23 this morning (5 hours ago) using version 0.12.0 and it has this same problem. I killed the kubelet and waited a couple of minutes, after which it was restarted and running happily (as reported by the master's /validate endpoint).
appears to be out of office the rest of this week and this is causing a significant number of cluster creation failures. Is there someone else who can own this and drive it to completion sooner?
cc/ Can you take a quick look at this?
what do you mean cluster creation failure? How big your cluster is? If you run 0.11 release, we should upgrade docker. also 's pr (() is not in 0.11 release. But 0.11 release doesn't have monit yet. We all observed sometimes docker daemon couldn't restart. With 0.11 that node is gone when docker failed to restart. For 0.12 release, monit is installed and configured to take care both docker and kubelet processes. pr is just making kubelet initialization failed when docker is not running. But both docker and kubelet are monitored by monit, eventually they are coming up. Things is definitely better than 0.11.
After 10 minutes the master's /validate endpoint is not reporting that all components are healthy 1 master, 1 node. I have only observed this on the 0.12.0 release.

My question about monit is whether is does anything more than keep the process running. I have observed that it will restart the kubelet if the process exists (restoring it to working order) but it seems content to leave a broken kubelet process running for hours, making the cluster unusable.
I will take a look and send a fix, assigning to myself. I believe that it only checks whether the PID is alive. I will file a separate issue for using /healthz or an alternative ().
It looks like monit is capable of health checking:
Monit just detects if process is running or not today, but we could config it to do so.
Please just handle issue introduced by DockerCache here. We are going to handle healthcheck and interval in separate issue.
+1 :)
Thanks a lot for taking this issue and sorry for introducing the problem.


Question: // Scan instances for groups we want open for _, instance := range allInstances { <del> securityGroupId := findSecurityGroupForInstance(instance) if isNilOrEmpty(securityGroupId) { </del> <ins> securityGroup, err := findSecurityGroupForInstance(instance, taggedSecurityGroups) if err != nil { return err } if securityGroup == nil { </ins> glog.Warning("Ignoring instance without security group: ", orEmpty(instance.InstanceId)) continue } <ins> id := aws.StringValue(securityGroup.GroupId) if id == "" { glog.Warningf("found security group without id: %v", securityGroup) continue } </ins> <del> instanceSecurityGroupIds[*securityGroupId] = true </del> <ins> instanceSecurityGroupIds[id] = true </ins> } // Compare to actual groups for _, actualGroup := range actualGroups { <del> if isNilOrEmpty(actualGroup.GroupId) { </del> <ins> actualGroupID := aws.StringValue(actualGroup.GroupId) if actualGroupID == "" { </ins> glog.Warning("Ignoring group without ID: ", actualGroup) continue } <del> actualGroupID := *actualGroup.GroupId </del> adding, found := instanceSecurityGroupIds[actualGroupID] if found && adding { // We don't need to make a change; the permission is already in place

If there are two security groups on an instance, we currently make an arbitrary choice. We should choose based on tags, as we normally do. For back-compat, we need to cope if there is one security group and it is not tagged. We should make sure kube-up tags our security groups, though it isn't really required for kube-up, but it is just good behaviour. cf


Question: } func TestProxyStop(t *testing.T) { <del> port, err := echoServer(t, "127.0.0.1:0") if err != nil { t.Fatal(err) } </del> lb := NewLoadBalancerRR() lb.OnUpdate([]api.Endpoints{{JSONBase: api.JSONBase{ID: "echo"}, Endpoints: []string{net.JoinHostPort("127.0.0.1", port)}}})

https://travis-
Looking.
Another instance, or a different thing? https://travis-


Question: }) }) <del> /* This end to end test is to simulate a scenario where after kubelet restart/node reboot application pods requesting devices appear before the device plugin pod exposing those devices as resources. The happy path is where after node reboot/ kubelet restart, the device plugin pod appears before the application pod. This PR and this e2e test aims to tackle the scenario where device plugin either does not appear first or doesn't get the chance to re-register itself. Since there is no way of controlling the order in which the pods appear after kubelet restart/node reboot, we can't guarantee that the application pod recovers before device plugin pod (the scenario we want to exercise here). If the device plugin pod is recovered before the test pod, we still can meaningfully reproduce the scenario by NOT sending the registration command. To do so sample device plugin is enhanced. For implementation details, refer to: `test/images/sample-device-plugin/sampledeviceplugin.go`. This enhancement allows auto-registration of the plugin to be controlled with the help of an environment variable: REGISTER_CONTROL_FILE. By default this environment variable is not present and the device plugin autoregisters to kubelet. For this e2e test, we use sample device plugin spec with REGISTER_CONTROL_FILE=/var/lib/kubelet/device-plugins/sample/registration to allow manual registeration of the plugin to allow an application pod (requesting devices) to successfully run on the node followed by kubelet restart where device plugin doesn't register and the application pod fails with admission error. Breakdown of the steps implemented as part of this e2e test is as follows: 1. Create a file `registration` at path `/var/lib/kubelet/device-plugins/sample/` 2. Create sample device plugin with an environment variable with `REGISTER_CONTROL_FILE=/var/lib/kubelet/device-plugins/sample/registration` that waits for a client to delete the control file. 3. Trigger plugin registeration by deleting the abovementioned directory. 4. Create a test pod requesting devices exposed by the device plugin. 5. Stop kubelet. 6. Remove pods using CRI to ensure new pods are created after kubelet restart. 7. Restart kubelet. 8. Wait for the sample device plugin pod to be running. In this case, the registration is not triggered. 9. Ensure that resource capacity/allocatable exported by the device plugin is zero. 10. The test pod should fail with `UnexpectedAdmissionError` 11. Delete the test pod. 12. Delete the sample device plugin pod. 13. Remove `/var/lib/kubelet/device-plugins/sample/` and its content, the directory created to control registration */ ginkgo.Context("With sample device plugin", func() { var deviceCount int = 2 var devicePluginPod *v1.Pod var triggerPathFile, triggerPathDir string // this test wants to reproduce what happened in https://github.com/kubernetes/kubernetes/issues/109595 ginkgo.BeforeEach(func(ctx context.Context) { ginkgo.By("Wait for node to be ready") gomega.Eventually(ctx, e2enode.TotalReady). WithArguments(f.ClientSet). WithTimeout(time.Minute). Should(gomega.BeEquivalentTo(1)) ginkgo.By("Setting up the directory and file for controlling registration") triggerPathDir = filepath.Join(devicePluginDir, "sample") if _, err := os.Stat(triggerPathDir); errors.Is(err, os.ErrNotExist) { err := os.Mkdir(triggerPathDir, os.ModePerm) if err != nil { klog.Errorf("Directory creation %s failed: %v ", triggerPathDir, err) panic(err) } klog.InfoS("Directory created successfully") triggerPathFile = filepath.Join(triggerPathDir, "registration") if _, err := os.Stat(triggerPathFile); errors.Is(err, os.ErrNotExist) { _, err = os.Create(triggerPathFile) if err != nil { klog.Errorf("File creation %s failed: %v ", triggerPathFile, err) panic(err) } } } ginkgo.By("Scheduling a sample device plugin pod") data, err := e2etestfiles.Read(SampleDevicePluginControlRegistrationDSYAML) if err != nil { framework.Fail(err.Error()) } ds := readDaemonSetV1OrDie(data) dp := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: SampleDevicePluginName, }, Spec: ds.Spec.Template.Spec, } devicePluginPod = e2epod.NewPodClient(f).CreateSync(ctx, dp) go func() { // Since autoregistration is disabled for the device plugin (as REGISTER_CONTROL_FILE // environment variable is specified), device plugin registration needs to be triggerred // manually. // This is done by deleting the control file at the following path: // `/var/lib/kubelet/device-plugins/sample/registration`. defer ginkgo.GinkgoRecover() framework.Logf("Deleting the control file: %q to trigger registration", triggerPathFile) err := os.Remove(triggerPathFile) framework.ExpectNoError(err) }() ginkgo.By("Waiting for devices to become available on the local node") gomega.Eventually(ctx, isNodeReadyWithSampleResources). WithArguments(f). WithTimeout(5 * time.Minute). Should(BeReady()) framework.Logf("Successfully created device plugin pod") devsLen := int64(deviceCount) // shortcut ginkgo.By("Waiting for the resource exported by the sample device plugin to become available on the local node") gomega.Eventually(ctx, isNodeReadyWithAllocatableSampleResources). WithArguments(f, devsLen). WithTimeout(5 * time.Minute). Should(HaveAllocatableDevices()) }) ginkgo.It("should deploy pod consuming devices first but fail with admission error after kubelet restart in case device plugin hasn't re-registered", func(ctx context.Context) { var err error podCMD := "while true; do sleep 1000; done;" ginkgo.By(fmt.Sprintf("creating a pods requiring %d %q", deviceCount, SampleDeviceResourceName)) pod := makeBusyboxDeviceRequiringPod(SampleDeviceResourceName, podCMD) testPod := e2epod.NewPodClient(f).CreateSync(ctx, pod) ginkgo.By("making sure all the pods are ready") err = e2epod.WaitForPodCondition(ctx, f.ClientSet, testPod.Namespace, testPod.Name, "Ready", 120*time.Second, testutils.PodRunningReady) framework.ExpectNoError(err, "pod %s/%s did not go running", testPod.Namespace, testPod.Name) framework.Logf("pod %s/%s running", testPod.Namespace, testPod.Name) ginkgo.By("stopping the kubelet") startKubelet := stopKubelet() ginkgo.By("stopping all the local containers - using CRI") rs, _, err := getCRIClient() framework.ExpectNoError(err) sandboxes, err := rs.ListPodSandbox(ctx, &runtimeapi.PodSandboxFilter{}) framework.ExpectNoError(err) for _, sandbox := range sandboxes { gomega.Expect(sandbox.Metadata).ToNot(gomega.BeNil()) ginkgo.By(fmt.Sprintf("deleting pod using CRI: %s/%s -> %s", sandbox.Metadata.Namespace, sandbox.Metadata.Name, sandbox.Id)) err := rs.RemovePodSandbox(ctx, sandbox.Id) framework.ExpectNoError(err) } ginkgo.By("restarting the kubelet") startKubelet() ginkgo.By("waiting for the kubelet to be ready again") // Wait for the Kubelet to be ready. gomega.Eventually(ctx, e2enode.TotalReady). WithArguments(f.ClientSet). WithTimeout(2 * time.Minute). Should(gomega.BeEquivalentTo(1)) ginkgo.By("making sure all the pods are ready after the recovery") var devicePluginPodAfterRestart *v1.Pod devicePluginPodAfterRestart, err = e2epod.NewPodClient(f).Get(ctx, devicePluginPod.Name, metav1.GetOptions{}) framework.ExpectNoError(err) err = e2epod.WaitForPodCondition(ctx, f.ClientSet, devicePluginPodAfterRestart.Namespace, devicePluginPodAfterRestart.Name, "Ready", 120*time.Second, testutils.PodRunningReady) framework.ExpectNoError(err, "pod %s/%s did not go running", devicePluginPodAfterRestart.Namespace, devicePluginPodAfterRestart.Name) framework.Logf("pod %s/%s running", devicePluginPodAfterRestart.Namespace, devicePluginPodAfterRestart.Name) ginkgo.By("Waiting for the resource capacity/allocatable exported by the sample device plugin to become zero") // The device plugin pod has restarted but has not re-registered to kubelet (as AUTO_REGISTER= false) // and registration wasn't triggered manually (by writing to the unix socket exposed at // `/var/lib/kubelet/device-plugins/registered`). Because of this, the capacity and allocatable corresponding // to the resource exposed by the device plugin should be zero. gomega.Eventually(ctx, isNodeReadyWithAllocatableSampleResources). WithArguments(f, int64(0)). WithTimeout(5 * time.Minute). Should(HaveAllocatableDevices()) ginkgo.By("Checking that pod requesting devices failed to start because of admission error") // NOTE: The device plugin won't re-register again and this is intentional. // Because of this, the testpod (requesting a device) should fail with an admission error. gomega.Eventually(ctx, getPod). WithArguments(f, testPod.Name). WithTimeout(time.Minute). Should(HaveFailedWithAdmissionError(), "the pod succeeded to start, when it should fail with the admission error") ginkgo.By("removing application pods") e2epod.NewPodClient(f).DeleteSync(ctx, testPod.Name, metav1.DeleteOptions{}, 2*time.Minute) }) ginkgo.AfterEach(func(ctx context.Context) { ginkgo.By("Deleting the device plugin pod") e2epod.NewPodClient(f).DeleteSync(ctx, devicePluginPod.Name, metav1.DeleteOptions{}, time.Minute) ginkgo.By("Deleting the directory and file setup for controlling registration") err := os.RemoveAll(triggerPathDir) framework.ExpectNoError(err) ginkgo.By("Deleting any Pods created by the test") l, err := e2epod.NewPodClient(f).List(context.TODO(), metav1.ListOptions{}) framework.ExpectNoError(err) for _, p := range l.Items { if p.Namespace != f.Namespace.Name { continue } framework.Logf("Deleting pod: %s", p.Name) e2epod.NewPodClient(f).DeleteSync(ctx, p.Name, metav1.DeleteOptions{}, 2*time.Minute) } ginkgo.By("Waiting for devices to become unavailable on the local node") gomega.Eventually(ctx, isNodeReadyWithoutSampleResources). WithArguments(f). WithTimeout(5 * time.Minute). Should(BeReady()) }) }) </del> }) func compareSRIOVResources(expected, got *sriovData) {

Failing: ` Failing since . Last passing: First failing:
The diff () shows that went in which looks related
/sig node
/kind regression /kind bug /triage accepted


Question: return defaultPC, nil } <del> func (p *priorityPlugin) getDefaultPriority() (int32, error) { </del> <ins> func (p *priorityPlugin) getDefaultPriority() (string, int32, error) { </ins> dpc, err := p.getDefaultPriorityClass() if err != nil { <del> return 0, err </del> <ins> return "", 0, err </ins> } if dpc != nil { <del> return dpc.Value, nil </del> <ins> return dpc.Name, dpc.Value, nil </ins> } <del> return int32(scheduling.DefaultPriorityWhenNoDefaultClassExists), nil </del> <ins> return "", int32(scheduling.DefaultPriorityWhenNoDefaultClassExists), nil </ins> }

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Description: If Pod did not set , admission controller will set to "default value". If the default value are from default priority class, it's better to also set of pod.
can I take this? /assign
, thanks :) I created a PR for functionality, but that'll be great if you can help add some e2e test for it. I can help to shepherd :)
thanks let me have a try :)


Question: "//pkg/kubelet/cm/cpuset:go_default_library", "//pkg/kubelet/cm/topologymanager:go_default_library", "//pkg/kubelet/cm/topologymanager/bitmask:go_default_library", <ins> "//pkg/kubelet/config:go_default_library", </ins> "//pkg/kubelet/container:go_default_library", "//pkg/kubelet/status:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_HTTPIngressPath = map[string]string{ "":         "HTTPIngressPath associates a path with a backend. Incoming urls matching the path are forwarded to the backend.", <del> "path":     "Path is matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/'. When unspecified, all paths from incoming requests are matched.", </del> <ins> "path":     "Path is matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/' and must be present when using PathType with value "Exact" or "Prefix".", </ins> "pathType": "PathType determines the interpretation of the Path matching. PathType can be one of the following values: * Exact: Matches the URL path exactly. * Prefix: Matches based on a URL path prefix split by '/'. Matching isn  done on a path element by element basis. A path element refers is then  list of labels in the path split by the '/' separator. A request is an  match for path p if every p is an element-wise prefix of p of then  request path. Note that if the last element of the path is a substringn  of the last element in request path, it is not a match (e.g. /foo/barn  matches /foo/bar/baz, but does not match /foo/barbaz).n* ImplementationSpecific: Interpretation of the Path matching is up ton  the IngressClass. Implementations can treat this as a separate PathTypen  or treat it identically to Prefix or Exact path types.nImplementations are required to support all path types. Defaults to ImplementationSpecific.", "backend":  "Backend defines the referenced service endpoint to which the traffic will be forwarded to.", }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I tried to use the default path I got . No error. According to the path is not needed. "When unspecified, all paths from incoming requests are matched." Create a ingress like so: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: brew Network plugin and version (if this is a network-related bug): N/A Others:
/sig network /area ingress It seems to be a problem of ingress and I suggest creating an issue in I find a similar one which is a little different there
We use - would this be a problem with the aws repo then?
/assign
So digging a bit here, the API docs needs to be improved. By the , is required if the PathPrefix is Exact or Prefix. To solve your issue specifically, you have two ways: Add that accepts ingress without paths Add and change PathPrefix to Prefix, so this becomes a catchAll I'll open a PR to fix the comment/docs. Thanks!
Thanks! In my case because I'm using ALB controller I had to use with prefix to get it working - see
/triage accepted PR will close this ;)


Question: // dockerFilter wraps around dockerfilters.Args and provides methods to modify // the filter easily. type dockerFilter struct { <del> f *dockerfilters.Args </del> <ins> args *dockerfilters.Args </ins> } func newDockerFilter(args *dockerfilters.Args) *dockerFilter { <del> return &dockerFilter{f: args} </del> <ins> return &dockerFilter{args: args} </ins> } func (f *dockerFilter) Add(key, value string) { <del> f.Add(key, value) </del> <ins> f.args.Add(key, value) </ins> } func (f *dockerFilter) AddLabel(key, value string) {

umbrella issue: kuberuntime uses to ]parse the version string of the api version]() from the runtime. When using with docker, it complains that Maybe we should just enforce that the api version returned by the runtime should be semver compatible. /cc
This is mentioned in . :) For now, we can add a after the docker apiversion to make it semver compatible.
Thanks for the reminder! We should make it clear in the API. This blocks kubelet from starting up, so we should fix it first.
I've checked the code, currently is only used in when checking the docker version compatibility. It is not exposed to the user by any means, so it should be fine to make some change ourselves, such as adding a . However, if we want to expose this to the user via in the future, it may make user confusing.


Question: <ins> Copyright 2011-2016 Canonical Ltd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. </ins>

we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


Question: } // Takes a list of strings and compiles them into a list of regular expressions <del> func CompileRegexps(regexpStrings StringList) ([]*regexp.Regexp, error) { </del> <ins> func CompileRegexps(regexpStrings []string) ([]*regexp.Regexp, error) { </ins> regexps := []*regexp.Regexp{} for _, regexpStr := range regexpStrings { r, err := regexp.Compile(regexpStr)

In case it isn't clear why this happened, it's because gcloud ships with an older version of kubectl (currently version 0.7.0). We keep it as up to date as we can, but it will always lag a release or so behind head.
Fixed by the revert.


Question: } return estimate, nil } <ins> // sortableGroupVersionResources sorts the input set of resources for deletion, and orders pods to always be last. // the idea is that the namespace controller will delete all things that spawn pods first in order to reduce the time // those controllers spend creating pods only to be told NO in admission and potentially overwhelming cluster especially if they lack rate limiting. type sortableGroupVersionResources []unversioned.GroupVersionResource func (list sortableGroupVersionResources) Len() int { return len(list) } func (list sortableGroupVersionResources) Swap(i, j int) { list[i], list[j] = list[j], list[i] } func (list sortableGroupVersionResources) Less(i, j int) bool { return list[j].Group == "" && list[j].Resource == "pods" } </ins>

I've noticed at least 10 separate test flake issues open in which at least one failure looks like this: Current list: &q=is%3Aissue%20label%3Akind%2Fflake%20is%3Aopen%20%22was%20not%20deleted%20within%20limit%22 The 10 current issues (note that some are 'broken test run' issues and might have multiple failure causes): cc
Just to clarify, the pods were deleted in the test (hence the message). I mentioned this before in
I was chasing down another p0 but now that that is resolved will try to look for more that we can do here. On Wednesday, July 27, 2016, Yu-Ju Hong wrote:
- why did you close this - this is happening quite frquently.
Thanks for reopening! We will now automatically choose and owner for auto-opened flake issues. I closed a bunch of old flakes that were created with no assignee. I want them to get recreated with an assignee. The hope is someone will actually look at these.
is another example of this flake.
has the same symptom too.
One more example: would it help if we bump the timeout from 5 min to N (e.g., 8) min? or should we ask tests that create significant number of pods to clean up after themselves?
I am going to try and look at optimizing the requeue behavior for tests that have a lot of pods and we are waiting for them to delete. Looks like once they delete, we don't observe their deletion until the next resync
Opened a PR to improve e2e framework ns deletion to avoid duplicate ns delete operations, and to give more detailed information on what content did not actually get deleted.


Question: srcs = [":package-srcs"], tags = ["automanaged"], ) <ins> go_test( name = "go_default_test", srcs = ["fake_pod_expansion_test.go"], embed = [":go_default_library"], deps = [ "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/client-go/testing:go_default_library", ], ) </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to use  in a unit test, however I get a nil pointer panic when is invoked. What you expected to happen: would return a that doesn't panic on any method call. Ideally, it would allow me to interact with the client in someway for testing. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: client-go version: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
I believe this also impacts returning logs via a reactor as does not implement
Seems to be a problem with . Since we're creating the request with all empty fields. Not sure what the best way it to resolve this. The best thing I came up with is to use to return a request with everything set up. But I'm a noob here so that's just a shot in the dark :man_shrugging:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The request returned has a nil which is used in the
I have filed a PR trying to fix this issue.


Question: webhook.WithExponentialBackoff(ctx, w.initialBackoff, func() error { result, err = w.tokenReview.CreateContext(ctx, r) return err <del> }) </del> <ins> }, webhook.DefaultShouldRetry) </ins> if err != nil { // An error here indicates bad configuration or an outage. Log for debugging. klog.Errorf("Failed to make webhook authenticator request: %v", err)

What would you like to be : The api server should retry sending audit events to configured webhook sinks regardless of the type of HTTP error code it might receive from the webhooks. Why is this needed: While some components may live with failed data transmissions, this is a particularly sensitive topic for the audit area. The API server audit mechanisms must ensure that no other reasonable option exists before dropping an audit event that may be the only trace of malicious behavior in the cluster. It seems that the current implementation may be improved further in this direction. Currently, when the API server is configured for non-blocking sending of audit events to a webhook audit sink backend, it will retry up to 5 times with exponential backoff, should there be an error in communicating with the webhook, as implemented . However, not all errors will trigger a retry. Some are considered transient and yield retry. Others simply stop the retry cycle. For example, it retries on receiving HTTP code , but not on receiving . The list of error codes that will yield a retry is actually quite small - three HTTP errors and one network error. Any other error will not trigger a retry and the audit events will be lost. The code comments explain that the retry will consider only recoverable failures, which may make sense in other cases, but as far as sending audit events to an audit backend webhook is concerned, it is not sufficient policy. A more robust approach would be to consider any error and give the backend service a chance to recover instead of relying on the error codes semantics and the webhook implementations interpretation of it. That is more inline with having the transmission of audit events as top priority. /sig auth /sig api-machinery


Question: func (p crashingExecProber) Probe(_ exec.Cmd) (probe.Result, string, error) { panic("Intentional Probe crash.") } <ins> func TestOnHoldOnLivenessCheckFailure(t *testing.T) { m := newTestManager() w := newTestWorker(m, liveness, api.Probe{SuccessThreshold: 1, FailureThreshold: 1}) status := getTestRunningStatus() m.statusManager.SetPodStatus(w.pod, getTestRunningStatus()) // First probe should fail. m.prober.exec = fakeExecProber{probe.Failure, nil} msg := "first probe" expectContinue(t, w, w.doProbe(), msg) expectResult(t, w, results.Failure, msg) if !w.onHold { t.Errorf("Prober should be on hold due to liveness check failure") } // Set fakeExecProber to return success. However, the result will remain // failure because the worker is on hold and won't probe. m.prober.exec = fakeExecProber{probe.Success, nil} msg = "while on hold" expectContinue(t, w, w.doProbe(), msg) expectResult(t, w, results.Failure, msg) if !w.onHold { t.Errorf("Prober should be on hold due to liveness check failure") } // Set a new container ID to lift the hold. The next probe will succeed. status.ContainerStatuses[0].ContainerID = "test://newCont_ID" m.statusManager.SetPodStatus(w.pod, status) msg = "hold lifted" expectContinue(t, w, w.doProbe(), msg) expectResult(t, w, results.Success, msg) if w.onHold { t.Errorf("Prober should not be on hold anymore") } } </ins>

Happened in GKE suite:
Does it need a liveness probe, maybe?
to delegate
Saw
Can you take a look at this one? Thanks!
FWIW, I saw some fishy messages in in (node: 104.197.48.84) It seems like docker was giving conflicting view of the container status, i.e., whether the container was running or not (and whether kubelet should kill it). kubelet's first attempt to kill the container : All the exec well after the grace period still shows that the container is the "Running: true" state () Later, kubelet tried killing the container again and failed because container does not exist. Note that after the failed kill also shows that the container was in the Running state. At the same time, cadvisor observed the container being stopped but then detected the container and then tried adding it again, but failed to find the /cc UPDATE: According to supervisord, docker was not killed during this time.
I didn't find result for the first Killing, does that really return? What is the log level, I remember that you told me before the log level in GKE is 2?
AFAICT, the pod worker may have never returned because the specific pod worker didn't log anything beyond that point. I was looking at (docker stop hangs) to see if it's relevant. didn't print any error though.
Yeah, the last SyncPod log was which is before the first Killing. Then the pod worker was stuck in
This really looks similar with I tried to reproduce it following the comment. The container was not actually killed. Docker stop hangs. Docker ps shows the container is still running. But the difference is that when I tried to kill it with another , it still hung there.
I was able to reproduce this quite easily by running 30 pods with the same spec in the test and with :
From
My theory is that during may cause docker to leave the container in a unrecoverable state (yep, a bug). I can consistently reproduce this by simply creating (which always fails), and leave the cluster for < 10 min. The restart counts of some of the pods will stop increasing after sometime, and after ssh'ing in to the node, one can easily confirm the same symptoms posted above. If pods use the http probe, they do not have this issue.

For the workaround, we can instruct the probe workers to stop probing after detecting a liveness check failure, until the container has been restarted (with a new container ID). Note that this is just a workaround, as it doesn't stop other components or users to run during this window. I've verified that with this fix, there are no zombie docker containers anymore. Will upload a PR.
FWIW, I also tested it with docker 1.8.3, which has the same problem.
Thanks !


Question: podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/kubelet" "k8s.io/kubernetes/test/e2e/framework" <del> e2ekubelet "k8s.io/kubernetes/test/e2e/framework/kubelet" </del> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2ewebsocket "k8s.io/kubernetes/test/e2e/framework/websocket" imageutils "k8s.io/kubernetes/test/utils/image"

What happened: In reviewing a test for promotion to conformance I noticed the kubelet API was being accessed via the API server's node proxy endpoint (ref: ). This seemed off, so I reviewed . It doesn't explicitly call out kubelet API access as allowed or disallowed. . and agreed kubelet API access seems like a bad idea. I'm now surveying what we have that does_ assume kubelet API access. It's definitely used by parts of e2e framework and some Conformance tests. I'm trying to verify how critical its use is. It appears that much of it is related to debugging or verifying whether kubelets and the control plane agree on who is running what. What you expected to happen: Find a blanket policy that bans kubelet API access for Conformance tests, whether directly, or indirectly through the API server's node proxy endpoint, at any time. Alternatively, find a policy that says kubelet API access may be allowed, but only for debugging or failure purposes (must not be on the critical path to make the test pass) Alternatively, find a policy that says kubelet API access is allowed, but only indirectly through the API server, and evaluated against the same constraints we use for control plane access (GA, non-optional) If we want to say kubelet API access is banned, we can use this as a tracking issue to demote whichever tests rely on it, and come up with a plan to verify the same behavior without kubelet API access. /area conformance /sig architecture /priority important-soon /milestone v1.16 /assign
Look for anything that ends up calling calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test requires kubelet API access to pass may end up calling it if the test fails, via: Some Conformance tests may end up calling this if the test fails specifically Other areas this is used: only if the test fails required for a pre-stop hook test to pass required to make sure the readonly kubelet apis aren't exposed via api server proxy resource usage tracking tests require kubelet api access to pass tests call this in
Looking for doesn't turn up any additional conformance tests, and I don't feel compelled to link the two other tests that use it as they are unlikely to ever be eligible.

The examples pointed out by on slack, the kubelet's endpoints is required to be available for these to pass
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
are there further changes for this targeted at 1.16?
Any update on this?
/milestone v1.17
Bug Triage Release Team checking in. Code freeze for the 1.17 release is on November 14. Will this make it in? Thanks!
/assign
hi is this still targetting for 1.17 or can we move to 1.18? Code freeze is Nov 14th. thanks!
/milestone v1.18
So we've updated the documented policy. My suggestion to close this out is to drop the [Conformance] tag from the tests that currently require the kubelet API to pass, and open issues that they need to be rewritten. Ignore whether the API is needed for diagnostic info in failure modes. SchedulerPredicates [Serial] calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test Pods should be submitted and removed [NodeConformance] requires kubelet API access to pass
i'd support this ^^
Hello , Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March. Is this issue still intended for milestone 1.18?
Hello There's been no movement on this issue this year. Should this get pushed to the next milestone?
yes, please John T Skarbek <:
/milestone v1.19
/cc


Question: } func (az *Cloud) getSecurityGroup() (nsg network.SecurityGroup, err error) { <ins> if az.SecurityGroupName == "" { return nsg, fmt.Errorf("securityGroupName is not configured") } </ins> securityGroup, err := az.nsgCache.Get(az.SecurityGroupName) if err != nil { return nsg, err

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: Created a service of type on Azure. The service is created successful and a LoadBalancer with public IP is created on Azure. kube-controller-manager goes into a crashloop What you expected to happen: LoadBalancer service receives public IP and is available for traffic How to reproduce it (as minimally and precisely as possible): provision a loadbalancer service on Azure with v1.10.3. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig api-machinery /sig azure
Logs of the initial request to create the load balancer service:
/remove-sig api-machinery
fyi
FYI
BTW, this is with an Availability Set.
Update: my was missing: Once I it LoadBalancer services work. Still, we shouldn't panic in the main loop on this.
Thanks I think we can close this issue. Create another for handling errors in both the cloud provider and the controller.
/cc
/assign
Filed to fix this


Question: "//vendor:k8s.io/apiserver/pkg/authentication/request/union", "//vendor:k8s.io/apiserver/pkg/authentication/request/x509", "//vendor:k8s.io/apiserver/pkg/authentication/token/tokenfile", <del> "//vendor:k8s.io/apiserver/pkg/authentication/user", </del> "//vendor:k8s.io/apiserver/plugin/pkg/authenticator/password/keystone", "//vendor:k8s.io/apiserver/plugin/pkg/authenticator/password/passwordfile", "//vendor:k8s.io/apiserver/plugin/pkg/authenticator/request/basicauth",

When an auth proxy tries to proxy an anonymous request, the user.Info ends up with the group system:authenticated, which is wrong.


Question: if framework.NodeOSDistroIs("windows") { privileged = false } <del> injectorPod, err := runVolumeTesterPod(f.ClientSet, config, "injector", privileged, fsGroup, tests) </del> <ins> injectorPod, err := runVolumeTesterPod(f.ClientSet, config, "injector", privileged, fsGroup, tests, false /*slow*/) </ins> if err != nil { framework.Failf("Failed to create injector pod: %v", err) return

Which jobs are flaking: gce-cos-master-alpha-features Which test(s) are flaking: [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] Testgrid link: Reason for failure: It takes 6 minutes to start a pod with PVC that has been restored from a snapshot, while the test waits only 5 minutes: csi-attacher sidecar says: Perhaps the provisioner should wait until the volume is ready to be used? That may not help with this test timeout, as PVC is provisioned after the Pod is scheduled and the total time could be the same. All this is caused by my PR to test block snapshots, It reworked the tests from using . If snapshots are slow, should all persistent storage tests use / ?
/sig storage
Generally, cutting a snapshot is quick, but uploading can take longer. Does provisioner wait until the snapshot is "ready" before returning success?
cc
(But yes, we should increase the timeout because overall time will still be longer no matter which order we take)
The provisioner does not wait. If snapshot is not readyToUse, creating a volume from it will fail. I'll have to check to see if there's logic to wait in the e2e test. Regarding GCE PD, at restore time, do we need to download the snapshot from object store? If that's the case, it could take a while to download as well.
volume controller should keep trying until snapshot is ready. Looks like attacher tried to attach while disk is not ready. It is strange because I think attach only happen after volume is successfully provisioned.
AFAIK, most cloud providers' snapshot ATM will be uploaded to some remote when taking, and will be downloaded when restoring. GCE PD is one of them.
It's not taking snapshots what slows the test, it's restoring it - returns quickly, but the returned volume is not really usable and attach fails for ~4 minutes until the snapshot is fully restored somewhere in GCE. Now that I am thinking about it, maybe it's better if the driver returns the volume early, so it does not end up orphaned when user deletes corresponding PVC and the provisioner is restarted at the same time.

Anyway, I'll extend the timeout in volume checks. /assign
Which brings me to idea: what about other storage backends? Some of them may take even longer that 10 minutes to restore a snapshot. Or more than 5 minutes to provision an empty volume and run a pod. There should be some configuration of timeouts (timeout multiplier?) for
I opened to investigate from GCE side if this behavior is expected.


Question: c.eventBroadcaster = broadcaster c.eventRecorder = recorder <del> c.readyCh = make(chan struct{}) </del> return c }

Verify Master - Master Blocking verify.openapi-spec Inconsistently since 2 November. First failure (shown in testgrid) API Server is not becoming ready during Relevant logs /sig-testing
cc for cidr controller
and may be related.
/sig auth network
/remove-sig auth Is almost certainly related
/assign it seems the servicecidr controller takes time to start and it blocks the apiserver readiness
/reopen Test run failed after related PR merger:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah /assign
/triage accepted
this is sig network can you pelase update the title?
/retitle [Flaking Test][sig-network] ci-kubernetes-verify-master verify-openapi-spec
This still flake in master blocking test grid.
use 60s timeout and here uses 30s timeout by default. It flakes. 10 lines of log cannot show other details. My local test with 15s timeout failed several times. 30s is enough for my local tests. Or we just enlarge the timeout for CI.
having both scripts the same timeout sounds reasonable I'm still worried why it keeps timing out, but for that we should first define a test that measure the apiserver startup and then decide what value is correct and enforce it.
/priority important-soon /milestone v1.30 backport to v1.29 may be needed.
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
This flakes in Do we have some action items for v1.30?
I need to fix it, is a shame on me, sorry
Hello! Release Signal shadow here. This issue has not been updated for a long time, so we'd like to check the status. The code freeze is starting less than a week, and we want to ensure that each PR has a chance to be merged on time.

As this issue is tagged for 1.30, is it still planned for this release? Please let us know the status if possible.
yeah, I like to get it fixed, there is some race condition that makes some of the boostrap to deadlock , I still didn't find why
didn't forget but didn't have time to figure out the root cause, will try to get back to this end of the week
I got one failure locally of an apiserver that took much time The apiservers need the servicecidr controller to start to be healthy, and the controllers need to sync the informer, but checking one of the requests from the informer I can see this error logging error output: "k8sx00nfnx02v1x12x06Statusx12x96x01nx06nx00x12x00x1ax00x12aFailurex1a2Timeout: Too large resource version: 2, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" and this keep going for a while logging error output: "k8sx00nfnx02v1x12x06Statusx12x98x01nx06nx00x12x00x1ax00x12aFailurex1a4Timeout: Too large resource version: 203, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" do you know why the resource version is growing?
looks like the server returns that error in a few places, not sure which one is getting exercised: update-openapi-spec starts with AllAlpha=true in order to enable all features / APIs ... are there any alpha features around watch / resourceVersion that could cause this?
Didn't read carefully the issue, so just quickly sharing some thoughts: 1) streaming lists might potentially be related, but it needs to be explicitly requested by the client.

Reflector is the only place that can currently do that, but it requires explicitly set env-var: Is it possible that this value is set? if so, then streaming lists will result is calling etcd for current RV and will wait for watchache to be synced with it - if progress-notify-interval is large and manual progress notify is disabled, we could face things like that ^^ 2) that said, if Alpha features are enabled, then manual-progress notify should also be set: 3) is it possible that we're facing the bug that Marek found 2 days before release that effectively was the main reason we didn't try to promote consistnet read to beta? ^^
Based on the log from I would guess this is the 3002ms implies the 3 second timeout of waiting for revision Empty implies consistent list from cache implies non-recursive list Planned to fix it, but PR was blocked on
thanks a ton for digging into that to deflake this test, I'd set this in the update-openapi- file (since it doesn't impact the generated openapi) with a comment to remove once is resolved


What would you like to be : Add a filter plugin that calls into EvenPodsSpread predicate, example Why is this needed: Part of /sig scheduling /help /priority important-soon
/assign
/remove-help
/retitle: [migration phase 1] EvenPodsSpread as filter plugin
/retitle [migration phase 1] EvenPodsSpread as filter plugin
/assign /unassign
May I take over this?
is going to take this over after the offline discussion, and he is familiar with EvenPodsSpread
IC, thanks a lot.


Question: framework.ExpectNoError(err, "Timeout while waiting to confirm PV %q deletion", retrievedPV.Name) }) <del> ginkgo.It("should apply changes to a pv/pvc status", func(ctx context.Context) { </del> <ins> /* Release: v1.29 Testname: PersistentVolumes(Claims), apply changes to a pv/pvc status Description: Creating PV and PVC MUST succeed. Listing PVs with a labelSelector MUST succeed. Listing PVCs in a namespace MUST succeed. Reading PVC status MUST succeed with a valid phase found. Reading PV status MUST succeed with a valid phase found. Patching the PVC status MUST succeed with its new condition found. Patching the PV status MUST succeed with the new reason/message found. Updating the PVC status MUST succeed with its new condition found. Updating the PV status MUST succeed with the new reason/message found. */ framework.ConformanceIt("should apply changes to a pv/pvc status", func(ctx context.Context) { </ins> pvClient := c.CoreV1().PersistentVolumes() pvcClient := c.CoreV1().PersistentVolumeClaims(ns)

[x] test approval issue : [x] test pr : [x] Flake [x] two weeks soak end date : 31 Oct 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 6 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 6 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/spf13/cobra" <ins> "k8s.io/kubernetes/pkg/api/latest" </ins> "k8s.io/kubernetes/pkg/kubectl" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" )

It would be nice if let me explain experimental resources to aid users ability to experiment.
It's possible this works today, but it was non-obvious how to use. Attempts:
cc
cc
No, this isn't supported, since current implementation only use v1 Client but not experimental Client.
do we need to fix this for 1.1?
How difficult do you think it would be to fix?
It has dependency on ServerAPIVersions(), for which we plan to make a clean-up PR. I guess it needs 3-4 days to get two PRs merged.
With , this can be fixed now. What's the expected behavior of ? I assume in case of , it should search hpa in both v1 and extensions/v1bata1, and determines hpa as a resource in the extensions group? BTW, the use of "api-version" flag is problematic It adds extra meaning to the generic "api-version" flag.


Question: - hostPath: path: /var/log name: varlog <ins> - hostPath: path: /run name: run - hostPath: path: /run/xtables.lock name: iptableslock </ins>

BUG REPORT Kubernetes version 2 Environment: Install tools: What happened: The code in takes care to supply the flag so it will wait on a lock to avoid interference between multiple processes modifying the table. If you read the source you discover it does this by creating a file . However, when is running in a container, is in a local filesystem, so it is not able to lock against processes running on the host or in other containers. And before you decide that the answer is to mount from the host, look at all the issues around leaking netns, e.g. Credit for pointing this out to me. Credit for pointing out locking functionality was to a few months back
Ironically, this is caused by a bugfix that moved to file-based locking. Previously, iptables would lock by reserving an abstract unix socket, which is tied to the network namespace.
Some discussion also in and following comments.
I experimented with this: but it didn't work on CoreOS, as the host wasn't using a version of iptables that used the file lock yet. Just ended up creating a directory call
I guess I should file this as an issue against kubeadm and any of the other sources of kube-proxy container config.
/assign
Should we leave this open (waiting for ... ?)
Waiting for I guess... (which I'm reviewing now)
i think this should not be marked as complete until the kubeadm fixes are in.
/reopen
/close fix in


Question: // the effects of previous updateFuncs to avoid potential conflicts. For example, if multiple // functions update the same field, updates in the last function are persisted. func (nim *nodeInfoManager) tryUpdateNode(updateFuncs ...nodeUpdateFunc) error { <ins> nim.lock.Lock() defer nim.lock.Unlock() </ins> // Retrieve the latest version of Node before attempting update, so that // existing changes are not overwritten.

What happened: I two CSI plugins to my Kubernetes cluster via a single call to . When I did this, the annotation was improperly set on 2 out of 3 nodes. Only one of the plugins appeared: What you expected to happen: Both and plugins should have appeared in the annotation. How to reproduce it (as minimally and precisely as possible): Apply two or more CSI plugins via a single call to . Anything else we need to know?: Kubelet logs indicate that it registered both plugins at the same time: The CSINode object looks complete: Only the node's annotation did not include Restarting kubelet fixed the issue. It looks like a race condition when InstallCSIDriver is called concurrently for multiple drivers. The code here: could lead to each thread observing the annotation as being empty, and then setting its own annotation that includes only itself. Environment: Kubernetes version (use ): v1.17.11 Cloud provider or hardware configuration: AWS OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): Linux ip-172-31-17-163 5.4.0-1021-aws -Ubuntu SMP Fri Jul 24 09:42:29 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: juju
/sig storage
/assign
I can reproduce this issue with commands below ! The correct csi nodeid would be like this


Question: } if curEndpoints.Has(ep) { <ins> // if we are syncing for the first time, loop through all current destinations and // reset their weight. if proxier.initialSync { for _, dest := range curDests { if dest.Weight != newDest.Weight { err = proxier.ipvs.UpdateRealServer(appliedVirtualServer, newDest) if err != nil { klog.ErrorS(err, "Failed to update destination", "newDest", newDest) continue } } } } </ins> // check if newEndpoint is in gracefulDelete list, if true, delete this ep immediately uniqueRS := GetUniqueRSName(vs, newDest) if !proxier.gracefuldeleteManager.InTerminationList(uniqueRS) {

Entry in ipvs real sever remains at weight=0 after kube-proxy restarts. (During 1.21 -1.22 kubernetes cluster upgrade.) Related to As the related issue is considered closed, I will report this issue again as the problem still persists in newer version. Might because of some corner cases. Real server weight should be 1 when endpoints is ready. Use kube-proxy in ipvs mode Deploy a service resource and its backend pods Make real sever not ready (or may need node not ready) so as to trigger graceful delete in kube-proxy Before kube-proxy deletes the entry from ipvs, restart kube-proxy Kube-proxy becomes ready and leave the weight=0 entry as it is Service is no longer accessible through cluster ip When accessed, dmesg shows kernel messge error ofDeletion of pod will trigger kube-proxy to remove weight=0 entry from ipvs so that after the entry is recreated service will become accessible If service is 1. LoadBalancer, and, 2. .spec.externalTrafficPolicy: Local, and, 3, service is accessed from loadbalancer IP, the entry in ipvs for loadbalancer IP will only have endpoints in the same node. If backends is DaemonSet, the only one endpoints will have weight=0 thus causing connection failure After kube-proxy restarts, endpoints will be processed in the following lines. Only when the entry is in the gracefuldeleteManager.InTerminationList will be deleted. Which is not the case if kube-proxy restarts before it deletes the ipvs entry. An existing entry in the ipvs is sufficient for endpoint to be considered healthy. Weight is not considered. kube-proxy should treat ipvs entry properly by the either the following two ways. kube-proxy should initiate InTerminationList and put ipvs entry with weight=0 into the termination list kube-proxy should also look for weight in the ipvs entry instead of only UniqueRSName <details</details<detailson-premises </details<details</details<details</details<details</details<details</details
/sig network
Graceful delete of ipvs is not only triggered when pod is in terminating phase. Graceful delete seems to be triggered when endpointslice status is ready=false. This will happen when node become not ready.

However, after the node become ready again, pods on the node will only be recreated if the duration of NotReady is longer than certain time. Otherwise the pod will just become ready again. If the node will not recreate the pods && kube-proxy set the pods to the graceful delete queue && kube-proxy restarts before deletion and all cache is cleared, kube-proxy will keep ipvs entry with weight=0 even if the real server is ready. Please correct me if I misunderstand something.
/area ipvs /area kube-proxy
Seems like a legit bug to me. I can't triage until wednesday though.
I can't recreate this when I try to follow thre instructions with pod not-ready rather than node not ready. When the pod is made ready again is not stated, so I tried; a connection to a pod it not-ready. The weight is set to 0, the connection prevails -start kube-proxy. The weight is still 0, connection still working the pod ready weight is set to 1 after a while, as it should
In the instruction is different, the pod is made ready while is away. That instruction shows the fault. The reproduce instruction in this issue is incorrect.
I triage this issue as a bug, but it's really a duplicate of which was never fixed (it seems). Don't know if the proper action is to close this issue and re-open . /triage accepted
Please instruct if this issue shall be closed as duplicate and re-opened PTAL at
Thank you for your triage Monitoring IPVS weights is costly, but needed while this problem persists. Any news? Adding the following lines should fix the problem but will also add loads.
That does seem costly to do on every endpoint sync. Is there an alternative solution we can explore where we assess all endpoints once when kube-proxy initially starts up?
This is a startup-only problem. It should be solved with a one-time action.
How about modifying to be ? We could then check and update the real server weights only when is true. Only and will be modified to call , all other users will call the function with false. Since these two functions, are only called during startup, we won't be doing this monitoring every endpoint sync.
cc:


Question: func TestPartialPermanentFailure(t *testing.T) { fake := &fakeDiscovery{ groupList: &metav1.APIGroupList{ <del> Groups: []metav1.APIGroup{{ Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }, { GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }}, </del> <ins> Groups: []metav1.APIGroup{ { Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }}, }, { Name: "astronomy2", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }, }, </ins> }, resourceMap: map[string]*resourceMapEntry{ "astronomy/v8beta1": {

Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: [sig-api-machinery] Garbage collector should not be blocked by dependency circle [Conformance] [sig-api-machinery] Garbage collector should orphan RS created by deployment when deleteOptions.PropagationPolicy is Orphan [Conformance] [sig-api-machinery] Garbage collector should support cascading deletion of custom resources [sig-api-machinery] Garbage collector should keep the rc around until all its pods are deleted if the deleteOptions says so [Conformance] [sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance] [sig-api-machinery] Garbage collector should delete pods created by rc when not orphaning [Conformance] [sig-api-machinery] Garbage collector should delete jobs and pods created by cronjob Testgrid link: Reason for failure: timeout waiting for deletion to complete Anything else we need to know: seems to have gotten much worse on/after 1/24/2020: See the time period on 2/11 for lots of examples number 2 flake according to /sig api-machinery
from the failure in gc events related to the test: test timed out: gc events related to the test:
cc
/assign thank you for fixing it!
top flake according to
looking at the debug logs from the last several failures, filtering remaining pods to name+finalizers+ownerReferences with: jq '[.[] | {"metadata":{"name","finalizers":(.metadata.finalizers // []), "ownerReferences": .metadata.ownerReferences}}]'
no consistent pattern between failures, GC just seems to have been running slower and sometimes timed out the test. tracked likely performance degradation to , fix in
note that should return performance to pre-1/24 levels, but this test was flaky even before that - *circle
Further debugging of GC lag issues in kube-controller-manager in e2e (after merged) Combined log of e2e+kcm+apiserver for the timeout test failure from is at The RC is deleted by the e2e test: GC starts running attemptToDelete on the rc and all pods within a second: attemptToDeleteItem on the RC (started at 17:37:28.)

takes ~17 seconds to process (end result is calling processDeletingDependentsItem on all the child pods): attemptToDelete on the pods (started at 17:37:28. - 17:37:28.) also takes ~17 seconds to complete (end result is GC making a DELETE API call on each pod): 2 seconds of ~17 second delay is the GC controller rebuilding its restmapper via discovery: The other 15 seconds are unaccounted for. Then, GC reacts to changing API group availability by performing its periodic resync of informers, which fails once (because another parallel e2e test removed a CRD at the same time), retries, takes a total of 39 seconds, and blocks item processing until complete: After resyncing monitors, it immediately adds all the items that received updates or were deleted while it was resyncing, including the 10 deleted pods (which no longer exist) and their parent RC: This time there is a ~16 seconds delay from attemptToDelete starting to handle the RC (at 17:38:26.) and the foreground deletion finalizer being removed: But the e2e test has already failed:
cc
The informer resync is not unexpected, because we have lots of parallel e2e tests that add/remove CRDs. The extra time for GC to recover from an API group sync failure (39 seconds, in this case) is not included in most of the GC e2e timeout periods. The 17 and 15 second execution times for attemptToDelete are worthy of more investigation. Combined, they made GC easily exceed the 60 second period (actual time: the test waits for the RC to be removed
adding fine-grained trace.LogIfLong statements to attemptToDelete in particular might be helpful in revealing exactly where we are spending so much time
Tried this out with The long delays outside of sync.Monitors are almost entirely due to the latency of I'm guessing that is fetching from discovery in this codepath for the same reason does: another parallel e2e test removed a CRD at the same time. more details:
Traced this down to the discovery 's cache refresh. See for a fix.


Question: WithStdinData("abcd1234n"). ExecOrDie() Expect(runOutput).ToNot(ContainSubstring("stdin closed")) <del> g := func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err := polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test-3", 1*time.Minute, g) if err != nil { os.Exit(1) } </del> <ins> g = func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err = polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test-3", 1*time.Minute, g) Expect(err).To(BeNil()) </ins> if !framework.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, time.Minute) { framework.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test-3") }

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: if !(runtime.IsMissingVersion(err) || runtime.IsMissingKind(err) || runtime.IsNotRegisteredError(err)) { return nil, err } <ins> // Only tolerate load errors if the file appears to be one of the two legacy plugin configs unstructuredData := map[string]interface{}{} if err2 := yaml.Unmarshal(data, &unstructuredData); err2 != nil { return nil, err } _, isLegacyImagePolicy := unstructuredData["imagePolicy"] _, isLegacyPodNodeSelector := unstructuredData["podNodeSelectorPluginConfig"] if !isLegacyImagePolicy && !isLegacyPodNodeSelector { return nil, err } </ins> // convert the legacy format to the new admission control format // in order to preserve backwards compatibility, we set plugins that // previously read input from a non-versioned file configuration to the

/kind bug What happened: When I want to enable both plugins by passing below flag, as they both need config file, and I find I donot know what to set If I set , then I get an error in apiserver
/kind bug /sig auth
/cc
/cc
first, you can't put comments in json. the error was complaining about your embedded comments. this would let the server start: second, the docs are really, really, not right (opened with fixes). if you want to specify config for multiple plugins, should point to a config that references individual config files for each plugin (which there's a bug in loading... fixed in ): example
/assign


Question: time.Sleep(10 * time.Millisecond) } <ins> count := reactor.getChangeCount() </ins> // Call the tested function err := test.test(ctrl, reactor, test) if err != nil { t.Errorf("Test %q initial test call failed: %v", test.name, err) } <del> reactor.waitTest() </del> <ins> for reactor.getChangeCount() < count+expectedChanges[ix] { reactor.waitTest() } </ins> evaluateTestResults(ctrl, reactor, test, t) }

Not many details in the logs: FAIL 2.721s
I ran tests a few times against recent master but couldn't see this, could it be an edgy case?
i'll look at it
should help a lot, still there will be a short window where the test could get wrong. Patch will follow when is merged.
Today I saw a very similar error to the one reported in , but seeing as that was closed as a dupe of this, adding details here:
this looks like something new. Let's keep this issue open for this unit test flake: I filled for integration test flake you reported.


Question: Description: A Pod is created with liveness probe on grpc service. Liveness probe on this endpoint should fail because of wrong probe port. When liveness probe does  fail then the restart count should +1. */ <del> ginkgo.It("should be restarted with a GRPC liveness probe [NodeConformance]", func(ctx context.Context) { </del> <ins> framework.ConformanceIt("should be restarted with a GRPC liveness probe [NodeConformance]", func(ctx context.Context) { </ins> livenessProbe := &v1.Probe{ ProbeHandler: v1.ProbeHandler{ GRPC: &v1.GRPCAction{

Instructions: Tests in : should not be restarted with a GRPC liveness probe should be restarted with a GRPC liveness probe /sig node /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue KEP:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Instructions: Tests in : /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : Instructions: Tests in : /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
/assign
/assign
Hi, I want to take this


Question: }, }, { <ins> name: "test with extender which filters out some Nodes", registerPlugins: []tf.RegisterPluginFunc{ tf.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), tf.RegisterFilterPlugin( "FakeFilter", tf.NewFakeFilterPlugin(map[string]framework.Code{"3": framework.Unschedulable}), ), tf.RegisterScorePlugin("NumericMap", newNumericMapPlugin(), 1), tf.RegisterBindPlugin(defaultbinder.Name, defaultbinder.New), }, extenders: []tf.FakeExtender{ { ExtenderName: "FakeExtender1", Predicates:   []tf.FitPredicate{tf.FalsePredicateExtender}, }, }, nodes:     []string{"1", "2", "3"}, pod:       st.MakePod().Name("test-filter").UID("test-filter").Obj(), wantNodes: nil, wErr: &framework.FitError{ Pod:         st.MakePod().Name("test-filter").UID("test-filter").Obj(), NumAllNodes: 3, Diagnosis: framework.Diagnosis{ NodeToStatusMap: framework.NodeToStatusMap{ "1": framework.NewStatus(framework.Unschedulable, `FakeExtender: node "1" failed`), "2": framework.NewStatus(framework.Unschedulable, `FakeExtender: node "2" failed`), "3": framework.NewStatus(framework.Unschedulable, "injecting failure for pod test-filter").WithPlugin("FakeFilter"), }, UnschedulablePlugins: sets.New("FakeFilter", framework.ExtenderName), }, }, }, { </ins> name: "test with filter plugin returning UnschedulableAndUnresolvable status", registerPlugins: []tf.RegisterPluginFunc{ tf.RegisterQueueSortPlugin(queuesort.Name, queuesort.New),

When Extender filters out some Nodes, we don't set any unschedulable plugins at all. It means Extender is completely ignored during the requeueing process. So, what's happening is: If Extender filters out all Nodes during scheduling, this Pod is soon retried because this Pod doesn't have any plugin name in unschedulable plugins. If Extender filters out some Nodes during scheduling and plugins filter out all other Nodes, this Pod is retried based on plugins' QueueingHint. Even if any cluster events happen and it could change Extender's decision (but not any plugins' decision), this Pod won't be requeued to activeQ/backoffQ. The latter case is serious because it could make Pods being stuck in unschedulable pod pool in 5min. We should have a short-term solution for the latter case. We can requeue Pods, which were rejected by Extender, by any kind of cluster events because we cannot know which events make Pods schedulable. Eventually, this issue makes us wonder how Pods rejected by Extender should be requeued. We cannot implement QHint equivalent in Extenders because it'd be too slow to call Extender every time any cluster events happen. Probably, somehow implementing EventsToRegister equivalent in Extender? Use Extender which does something in Filter. No response master <detailsn/a </details<details</details<details</details<details</details<details</details
/assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/reopen We'll close it when all cherry-picks are done.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close Cherry-picks are done.