Question: func TestWaitForPathToExist(t *testing.T) { devicePath := []string{"/dev/disk/by-path/ip-127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0", <del> "/dev/disk/by-path/pci-0000:00:00.0-ip-127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0"} </del> <ins> "/dev/disk/by-path/pci-*-ip-127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0"} fpath := "/dev/disk/by-path/pci-0000:00:00.0-ip-127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0" </ins> <del> exist := waitForPathToExistInternal(devicePath[0], 1, "tcp", fakeOsStat, filepath.Glob) </del> <ins> exist := waitForPathToExistInternal(&devicePath[0], 1, "tcp", fakeOsStat, filepath.Glob) </ins> if exist == false { t.Errorf("waitForPathToExist: could not find path %s", devicePath[0]) } <del> exist = waitForPathToExistInternal(devicePath[0], 1, "fake_iface", fakeOsStat, filepath.Glob) </del> <ins> exist = waitForPathToExistInternal(&devicePath[0], 1, "fake_iface", fakeOsStat, filepath.Glob) </ins> if exist != false { t.Errorf("waitForPathToExist: wrong code path called for %s", devicePath[0]) } <del> exist = waitForPathToExistInternal(devicePath[1], 1, "fake_iface", os.Stat, fakeFilepathGlob) </del> <ins> exist = waitForPathToExistInternal(&devicePath[1], 1, "fake_iface", os.Stat, fakeFilepathGlob) </ins> if exist == false { t.Errorf("waitForPathToExist: could not find path %s", devicePath[1]) } <del> exist = waitForPathToExistInternal(devicePath[1], 1, "tcp", os.Stat, fakeFilepathGlob) </del> <ins> exist = waitForPathToExistInternal(&devicePath[1], 1, "tcp", os.Stat, fakeFilepathGlob) </ins> if exist != false { t.Errorf("waitForPathToExist: wrong code path called for %s", devicePath[1]) } <ins> exist = waitForPathToExistInternal(&devicePath[1], 1, "fake_iface", os.Stat, fakeFilepathGlob2) if devicePath[1] != fpath { t.Errorf("waitForPathToExist: wrong code path called for %s", devicePath[1]) } </ins> }

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-beta.1", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2017-06-08T13:08:32Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare-Metal, HP SE316M, 2x X5675, 144GB, iSCSI/RDMA/iSer OS: CentOS 7 Kernel: Linux at-host-18 4.9.31 SMP Wed Jun 7 22:40:40 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: What happened: I want to mount a iSCSI volume via iSER (RDMA over Infiniband) in this case the iSCSI iface is "iser" and not tcp, therefore the glob path for device mount is selected (), which then is passed to the fsck and mount commands without globbing the path (e.g. calling filepath.Glob) Therefore the iSCSI volume is not mounted and the pod cannot be started. Logs: What you expected to happen: To successfully mount the volume. How to reproduce it (as minimally and precisely as possible): Setup an iSCSI via RDMA (iSER) environment without multipath and try to mount a volume for a pod. Anything else we need to know: I believe it's just needed to add the globbing at line 259 like so: devicePaths = append(devicePaths, filepath.Glob(devicePath)) As I'm not a go programmer and have no experience with it, probably somebody could create a commit which I would gladly test!
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team.

You can find the team list and label list
/sig storage
I'll take a look.
/sig storage
on a local build that fixed the issue: (e.g. it compiles)
I don't want to add filepath.Glob to that general path because that line is also executed "tcp" case. I'm considering the fix below. Pretty similar.
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. )
Can we suppose that filepath.Glob(devicePath) always only return one device path if transport is "tcp"?
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. ) Thank you for above comment. I updated the logic. Could you confirm the fix on the PR?
Pretty sure that tcp transport will always only have one path - and it will exist (there's no glob in the check path)


Question: } } <ins> func TestEndpointSliceMirroringSelectorTransition(t *testing.T) { controlPlaneConfig := framework.NewIntegrationTestControlPlaneConfig() _, server, closeFn := framework.RunAnAPIServer(controlPlaneConfig) defer closeFn() config := restclient.Config{Host: server.URL} client, err := clientset.NewForConfig(&config) if err != nil { t.Fatalf("Error creating clientset: %v", err) } resyncPeriod := 12 * time.Hour informers := informers.NewSharedInformerFactory(client, resyncPeriod) epsmController := endpointslicemirroring.NewController( informers.Core().V1().Endpoints(), informers.Discovery().V1().EndpointSlices(), informers.Core().V1().Services(), int32(100), client, 1*time.Second) // Start informer and controllers stopCh := make(chan struct{}) defer close(stopCh) informers.Start(stopCh) go epsmController.Run(1, stopCh) testCases := []struct { testName               string startingSelector       map[string]string startingMirroredSlices int endingSelector         map[string]string endingMirroredSlices   int }{ { testName:               "nil -> {foo: bar} selector", startingSelector:       nil, startingMirroredSlices: 1, endingSelector:         map[string]string{"foo": "bar"}, endingMirroredSlices:   0, }, { testName:               "{foo: bar} -> nil selector", startingSelector:       map[string]string{"foo": "bar"}, startingMirroredSlices: 0, endingSelector:         nil, endingMirroredSlices:   1, }, { testName:               "{} -> {foo: bar} selector", startingSelector:       map[string]string{}, startingMirroredSlices: 1, endingSelector:         map[string]string{"foo": "bar"}, endingMirroredSlices:   0, }, { testName:               "{foo: bar} -> {} selector", startingSelector:       map[string]string{"foo": "bar"}, startingMirroredSlices: 0, endingSelector:         map[string]string{}, endingMirroredSlices:   1, }, } for i, tc := range testCases { t.Run(tc.testName, func(t *testing.T) { ns := framework.CreateTestingNamespace(fmt.Sprintf("test-endpointslice-mirroring-%d", i), server, t) defer framework.DeleteTestingNamespace(ns, server, t) meta := metav1.ObjectMeta{Name: "test-123", Namespace: ns.Name} service := &corev1.Service{ ObjectMeta: meta, Spec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{{ Port: int32(80), }}, Selector: tc.startingSelector, }, } customEndpoints := &corev1.Endpoints{ ObjectMeta: meta, Subsets: []corev1.EndpointSubset{{ Ports: []corev1.EndpointPort{{ Port: 80, }}, Addresses: []corev1.EndpointAddress{{ IP: "10.0.0.1", }}, }}, } _, err = client.CoreV1().Services(ns.Name).Create(context.TODO(), service, metav1.CreateOptions{}) if err != nil { t.Fatalf("Error creating service: %v", err) } _, err = client.CoreV1().Endpoints(ns.Name).Create(context.TODO(), customEndpoints, metav1.CreateOptions{}) if err != nil { t.Fatalf("Error creating endpoints: %v", err) } // verify the expected number of mirrored slices exist err = waitForMirroredSlices(t, client, ns.Name, service.Name, tc.startingMirroredSlices) if err != nil { t.Fatalf("Timed out waiting for initial mirrored slices to match expectations: %v", err) } service.Spec.Selector = tc.endingSelector _, err = client.CoreV1().Services(ns.Name).Update(context.TODO(), service, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Error updating service: %v", err) } // verify the expected number of mirrored slices exist err = waitForMirroredSlices(t, client, ns.Name, service.Name, tc.endingMirroredSlices) if err != nil { t.Fatalf("Timed out waiting for final mirrored slices to match expectations: %v", err) } }) } } func waitForMirroredSlices(t *testing.T, client *kubernetes.Clientset, nsName, svcName string, num int) error { t.Helper() return wait.PollImmediate(1*time.Second, wait.ForeverTestTimeout, func() (bool, error) { lSelector := discovery.LabelServiceName + "=" + svcName lSelector += "," + discovery.LabelManagedBy + "=endpointslicemirroring-controller.k8s.io" esList, err := client.DiscoveryV1().EndpointSlices(nsName).List(context.TODO(), metav1.ListOptions{LabelSelector: lSelector}) if err != nil { t.Logf("Error listing EndpointSlices: %v", err) return false, err } if len(esList.Items) != num { t.Logf("Expected %d slices to be mirrored, got %d", num, len(esList.Items)) return false, nil } return true, nil }) } </ins> // isSubset check if all the elements in a exist in b func isSubset(a, b map[string]string) bool { if len(a) > len(b) {

with Selector is created is removed Endpoints and EndpointSlice controllers do not clean up Endpoint(Slice)s mirroring controller starts mirroring the Endpoints resource originally created by the Endpoints controller is back to the Service backing Service change created by mirroring controller still references original IP from step 4 At least one of the following: Endpoints and EndpointSlices to be cleaned up when the Service has the selector removed () mirrored EndpointSlice to be removed when the Service has a selector back See "What Happened" No response Most noticeable in Kubernetes 1.19+ since that's when kube-proxy starts reading from EndpointSlices by default. Any No response No response No response No response
/sig network /triage accepted /priority important-soon /cc
/assign


Question: testpatterns.FileSizeLarge:  "8d763edc71bd16217664793b5a15e403", } <ins> const mountPath = "/opt" </ins> type volumeIOTestSuite struct { tsInfo TestSuiteInfo }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow] Since when has it been failing: 11/13 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: This is consistently failing in 1.13 branch and is release blocking. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test /sig storage /priority critical-urgent /milestone v1.13 /assign /cc
The test case is failing because it wrote 2MB instead of the expected 1MB. It's also very odd that the test is failing in soak, but passing in regular CI suites.
There's a few problems: .GetVolumeSource returns "/tmp" for everything. This means all tests will share the same directory. This should be probably use some unique directory instead volume IO test generates a unique mountPath directory, however that doesn't mean a unique directory is actually created on the host. Ie, a mountPath of "/a/b/c" on a hostpath volume at "/tmp" doesn't mean directory "/tmp/a/b/c" is created. io test file name is not unique seems like the IO test is not cleaning up the files it creates, so the files are getting reused on subsequent tests due to the non-unique naming issues.
The other thing I can't explain is why it's failing only now
Fix is here:
As per discussion in 1.13 burndown this morning, this is a test job issue and its not release blocking at this point. So we can take this fix in 1.14 as opposed now during code freeze let us know if you feel otherwise
/milestone v1.14 /priority important-soon
/remove-priority critical-urgent


Question: mountName := fmt.Sprintf("path%d", i) volumes[i].Name = mountName if src == "" { <del> volumes[i].VolumeSource.EmptyDir = &v1.EmptyDirVolumeSource{Medium: v1.StorageMediumMemory} </del> <ins> volumes[i].VolumeSource.EmptyDir = &v1.EmptyDirVolumeSource{} </ins> } else { volumes[i].VolumeSource.HostPath = &v1.HostPathVolumeSource{ Path: src,

Noticing this is in our gce 5k-node cluster tests. It has been continuously failing for 4 runs now already: https://k8s- https://k8s- (and 39 and 40) The e2e test concerned is - Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content And it is failing due to: cc
cc - Could one of you PTAL or redirect as apt?
FTR, it is passing on small scale - https://k8s- I took a quick look at the and the following PR catches my attention at first sight: test: e2e: support NFS test on overlayfs
Do we have some umbrella issue for failing correctness tests on large scale atm?
Not really, we have , though.
does look very suspicious. Before, the NFS server test image would use the container image layer for its data, and this change makes it use emptydir tmpfs instead. for the large scale tests, do you guys have default memory limits for pods? That could potentially explain why it's failing on large scale tests and not the regular tests. A possible fix could be to change it to use regular emptydir so it's not subject to any memory limits.
FMU we're not setting any memory limits for pods explicitly and it's the same e2e test running on both the large cluster and the small cluster. I'm not sure if there's sth (like an admission plugin) which secretly sets memory limits for pods based on cluster size - I'd be quite surprised if that's the case.
Hmm or it could be that in the scale test, the nodes are more loaded, so resources could be more constrained. In any case, I think it's better for this test to not write to tmpfs, because it can write gigs of data.
I would be surprised a bit if that was the case - as this test was passing for quite some time (https://k8s-). So mostly unless sth changed around this e2e test or with storage/kubernetes, it doesn't seem to be so. Anyhow, not writing gigs of data to tmpfs does sound good to me :) One more thing to note is we're using the following config for our large cluster:
Yah so this test got changed from using disk to using memory by , which is when this test started failing.

So I suspect it is related to either a memory limit or memory pressure.
That seems to have . Thanks
More references FTR: https://k8s- https://k8s- https://k8s-


Question: if err != nil { return nil, err } <ins> for i, ing := range service.Status.LoadBalancer.Ingress { if ing.IP == "" && ing.Hostname == "" { return nil, fmt.Errorf("unexpected Status.LoadBalancer.Ingress[%d] for service: %#v", i, ing) } } </ins> return j.sanityCheckService(service, v1.ServiceTypeLoadBalancer) }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should handle load balancer cleanup finalizer for service [sig-network] Services should only allow access from service loadbalancer source ranges Since when has it been failing: 2019-10-19 Testgrid link: https://k8s- Reason for failure: and Anything else we need to know: /sig scalability /sig network could you triage?
It seems that new test was at that point: [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout My personal guess is that we started doing too much load-balancer-related stuff at the same time.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think this is my fault. should fix it. Is there some way to those tests to run against a PR? /remove-triage unresolved
At the moment the fastest way would be to merge your fix and wait for the next run tomorrow.
Reopening just to keep this tracking - will close once we have green run.
the fix helped for a two tests, but one of them is still failing: https://k8s-
I disagree - there are couple loadbalancing-related tests that are flaking with high flakiness ration - I think this PR didn't change anything. So my feeling is that is may still be something like:
Still, the conclusion is that unfortunately more work in needed to put us back in a good shape.
I looked a bit into it and the conclusions are: "Services should handle load balancer cleanup finalizer for service [Slow]" test is consistently failing in small clusters too with: IIUC, the reason is that if we downgrade load-balancer service to non-loadbalaced one, it takes some time to remove the ingress from Status, but it is failing immediately due to this: - can you please take a look? the apps test (and I think couple more) are failing due to incorrect timeouts - I will send out a fix for it soon
Actually, this one will be still failing as of now - the reason for it is described here:

Which jobs are failing: Which test(s) are failing: Since when has it been failing: 18 Oct 13:17 PDT Testgrid link: https://k8s- Reason for failure: check
/sig network /cc /cc
/priority critical-urgent /milestone v1.17
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I believe this is caused by this merge cc.
Most of this is caused by and should be fixed by . Is there any way to run these tests against that PR? /remove-triage unresolved
The job is still failing. Can you check?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage unresolved
OK, fixed some of the tests, got the rest to the point where they failed differently...
Thanks for the update. In testgrid we still see the same error. We will keep observing the next runs.
- still failing. This time the error is We probably need to clean up sanityChecks a bit more - this seems to be a similar issue.
yeah, I'll temporarily un- the tests in my next PR so I can make sure they're actually fixed this time
After the last fix, it seemed to finally recover.


Question: // GetDatastoresInZone returns all the datastores in the specified zone func (nm *NodeMapper) GetDatastoresInZone(vc string, zone string) []string { <del> return vcToZoneDatastoresMap[vc][zone] </del> <ins> nm.nodeInfoRWLock.RLock() defer nm.nodeInfoRWLock.RUnlock() return nm.vcToZoneDatastoresMap[vc][zone] </ins> } // GetNodeInfo returns NodeInfo for given nodeName func (nm *NodeMapper) GetNodeInfo(nodeName string) *NodeInfo { <del> return nameToNodeInfo[nodeName] </del> <ins> nm.nodeInfoRWLock.RLock() defer nm.nodeInfoRWLock.RUnlock() return nm.nameToNodeInfo[nodeName] </ins> } // SetNodeInfo sets NodeInfo for given nodeName. This function is not thread safe. Users need to handle concurrency. func (nm *NodeMapper) SetNodeInfo(nodeName string, nodeInfo *NodeInfo) { <del> nameToNodeInfo[nodeName] = nodeInfo </del> <ins> nm.nodeInfoRWLock.Lock() defer nm.nodeInfoRWLock.Unlock() nm.nameToNodeInfo[nodeName] = nodeInfo </ins> }

Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted


Question: "k8s.io/apimachinery/pkg/util/uuid" utilversion "k8s.io/apimachinery/pkg/util/version" clientset "k8s.io/client-go/kubernetes" <ins> podutil "k8s.io/kubernetes/pkg/api/v1/pod" </ins> "k8s.io/kubernetes/test/e2e/framework" e2eevents "k8s.io/kubernetes/test/e2e/framework/events" e2ekubelet "k8s.io/kubernetes/test/e2e/framework/kubelet"

What happened: I am questioning validity of the test. Comments under of say: The test scenario described as it is sounds reasonable. The only nitpick I have is the very first sentence saying "". Pod's resource limits are ignored in predicates. They take effect in the scoring phase when priorities are computed. Also, priorities are computed only when there are at least two candidate nodes. So after revisiting the step number 5 it's impossible to make sure the additional pod does or does not get scheduled when setting the resource limits. There are either no candidate nodes or one. In which case the limits are ignored and the test passes in the first case and fails in the second. In case there are at least two candidate nodes, one of them will always get picked. In which case the additional pod gets scheduled (no matter what the resource limits field says). In the framework: resource limits are accessed through method [1] is used to perform pre-scoring [2] in the scoring phase score is computed through method [3] The test is alternating between passing and failing state. It passes when predicates give empty list of candidate nodes and failing when the list of candidate nodes is non-empty. Thus, making the pod resource limits ineffective. [1] [2] [3] What you expected to happen: Have another node schedule the additional pod validating that all saturated nodes (with scheduled) have lower score (0 in case of plugin). Or, invalidate the test by removing it. Can anyone check the test validity and confirm/disprove my statement about ineffectiveness of the resource limits? How to reproduce it (as minimally and precisely as possible): Make sure there is at least one node with non-zero free cpu resource before running the step 5. Anything else we need to know?: Environment: Kubernetes version (use ): master HEAD Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
cc
I have noticed you refactored the test in Can you still recall the original intent of the test? I can't grasp how the limits are actually taken into account.
I agree, I think the test is wrong, the scheduler doesn't make fit decisions on resource limits. In the test, should have set. do you mind fixing the test and update the test comment please?
On it /assign


Question: return clientConfig } <ins> // toRawKubePersistentConfigLoader binds config flag values to config overrides // Returns a persistent clientConfig for propagation. func (f *ConfigFlags) toRawKubePersistentConfigLoader() clientcmd.ClientConfig { f.lock.Lock() defer f.lock.Unlock() if f.clientConfig == nil { f.clientConfig = f.toRawKubeConfigLoader() } return f.clientConfig } </ins> // ToDiscoveryClient implements RESTClientGetter. // Expects the AddFlags method to have been called. // Returns a CachedDiscoveryInterface using a computed RESTConfig.

/kind bug What happened: takes ~11 seconds when there are thousands of CRDs and is around 28KiB. logs the following message once per resource: Removing unnecessary entries from reduces the runtime by around ten seconds. What you expected to happen: only parses once, and therefore doesn't run unnecessarily slowly. How to reproduce it (as minimally and precisely as possible): shows the config being loaded multiple times (assuming you have multiple pods in the current cluster). Alternatively, add 10-15 clusters to and create ~3000 of a cheap resource (eg custom resources or empty config maps). Run on this resource and observe that it is unnecessarily slow. Anything else we need to know?: I've created a CPU profile using the latest CI build of kubectl, which shows ~12 seconds being spent in . Environment: Kubernetes version (use ): v1.12.1 (also Cloud provider or hardware configuration: GKE or minikube --vm-driver=none OS (e.g. from /etc/os-release): Debian Buster-like Kernel (e.g. ): 4.17.0
/sig cli
/assign
Same problem here. I have less resources but more contexts in my kubeconfig file. With 90 pods and ~300 contexts in the kubeconfig it also takes ~10 seconds for a simple kubectl get pods Note: I tried it with kubectl: 1.11.0, 1.12. With kubectl 1.10.5 the kubeconfig was loaded only once.
I'm unable to reproduce this as of the latest master. My kubeconfig is only loaded 4 times, no matter how many resources are loaded.
When I try a kubectl v1.13.0-alpha.3 against unfortunately a 1.9.11 Kubernetes Cluster I still have the problem: `
Maybe the bug is Linux-only? Presumably the kubeconfig location is different on different platforms, so maybe it has a different code path. I can repro on latest master:
I filed to fixed this. You can try to help validate it. Thanks.
Works for me, thanks!


Question: type DesiredStateOfWorldPopulator interface { Run(sourcesReady config.SourcesReady, stopCh <-chan struct{}) <del> // ReprocessPod removes the specified pod from the list of processedPods // (if it exists) forcing it to be reprocessed. This is required to enable </del> <ins> // ReprocessPod sets value for the specified pod in processedPods // to false, forcing it to be reprocessed. This is required to enable </ins> // remounting volumes on pod updates (volumes like Downward API volumes // depend on this behavior to ensure volume content is updated). ReprocessPod(podName volumetypes.UniquePodName)

It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!


Question: selfLinkPrefix := "" switch crd.Spec.Scope { case apiextensions.ClusterScoped: <del> selfLinkPrefix = "/" + path.Join("apis", crd.Spec.Group, crd.Spec.Version) + "/" </del> <ins> selfLinkPrefix = "/" + path.Join("apis", crd.Spec.Group, crd.Spec.Version) + "/" + crd.Status.AcceptedNames.Plural + "/" </ins> case apiextensions.NamespaceScoped: selfLinkPrefix = "/" + path.Join("apis", crd.Spec.Group, crd.Spec.Version, "namespaces") + "/" }

Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a custom resource definition with -this is important - scope "Cluster". Then I created an element of that kind. Then I fetched the element and saw that the "metadata.selfLink" attribute had an incorrect value. The path did not contain the custom resource name. Trying to fetch the element using API results in 404 response. What you expected to happen: The "metadata.selfLink" of an instance of a custom resource should be correct. How to reproduce it (as minimally and precisely as possible): Prepare a custom resource definition with scope: Cluster (important! with scope Namespaced the selfLink is correct) in a file (slightly modified standard example from here: ) Create it: Now prepare a custom object in a file my- with this content: And create it: Now get the custom object list: The result is: As you can see the selfLink is . This is wrong, it should be . Using the selfLink as it is causes 404 when fetching the resource via API. Anything else we need to know?: This only happens when the custom resource definition has scope: Cluster. Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: 2017 Macbook Pro OS (e.g. from /etc/os-release): macOS High Sierra 10.13.3 Kernel (e.g. ): Darwin Kernel Version 17.4.0 Install tools: minikube 0.24.1
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Created the PR:
/cc


Question: // enough space as per the garbage collection policy. GarbageCollect() error <ins> // Start async garbage collection of images. Start() error </ins> // TODO(vmarmol): Have this subsume pulls as well. }

I'm trying to start 100-node cluster. I got the following error in kubelet: E0505 08:31:55. 5066 ] Failed to create kubelet: failed to initialize image manager: failed to start image manager: dial unix connection refused After ssh-ing I checked that docker is working by running . I guess it must be some kind of a race where kubelet starts before docker is ready. When I start 3-node cluster it works just fine. This happened to me 3 times in a row. Please add this to 1.0 milestone if necessary.
I've just exactly the same problem (when creating a 100-node cluster)
This is a bug in our initialization logical. We initialize image manager which needs to talk to docker at very early stage before docker is up yet. Later kubelet does a wait to make sure docker is up. We should either delay the initialization of image manager or handle docker unavailable better. Kubelet will be restarted by monit eventually, but this definitely slow down node initialization. I mark this for 1.0. Should be a simple fix if we just simply delay the initialization of image manager.
I'll go ahead and fix it (since I am mainly to blame :) )
Changed to P0 because asked us to only add P0 issues to v1.0 milestone.


Question: return r.finalize(service, slicesToCreate, slicesToUpdate, slicesToDelete, triggerTime) } <ins> // placeholderSliceCompare is a conversion func for comparing two placeholder endpoint slices. // It only compares the specific fields we care about. var placeholderSliceCompare = conversion.EqualitiesOrDie( func(a, b metav1.OwnerReference) bool { return a.String() == b.String() }, func(a, b metav1.ObjectMeta) bool { if a.Namespace != b.Namespace { return false } for k, v := range a.Labels { if b.Labels[k] != v { return false } } for k, v := range b.Labels { if a.Labels[k] != v { return false } } return true }, ) </ins> // finalize creates, updates, and deletes slices as specified func (r *reconciler) finalize( service *corev1.Service,

My cluster contains about 4w endpointslices. There are 3 related metrics (endpointslicecontrollerendpointsaddedpersynccount, endpointslicecontrollerendpointsremovedpersynccount, endpointslicecontrollerchanges) to indicates endpoint add remove and update. I found that the apiserver periodically receives lots of endpointslice update requests. In terms of the number of requests, the endpointslice has been fully updated. But from the endpointslice resource itself, it seems that it has not been updated, because "" annotation didn't change It seems that there is no need to update the endpointslice with no really changes. On the other hand, a large number of ep slice update requests will stress the apiserver. In order to make the phenomenon more obvious, the min-resync-period parameter of controller-manager can be adjusted to 1 minute. Create one or more empty endpointslices, and then you can find in the apiserver logs that requests to update these endpointslice every one to two minutes. For clusters with lots of empty endpointslices, this bug may affect the stability of it. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
there were different improvements in this area, but since 1.20 is going EOL in 8 days it will be good to know if this happens in 1.21 /cc
I'm glad to recieve your comment. I'm not sure because I only have cluster with version 1.20.10. Expect someone who could provide more detail about it. It is actually happening in my cluster, but I don't know how it happens.
I've seen this in 1.20 clusters as well, but have not seen it in 1.21.

My theory was that this is mostly or entirely fixed by Prior to that, any change to EndpointSlice labels would not trigger an EndpointSlice controller sync loop, but the resync period would result in those EndpointSlices being updated at that point. more recent fix () seems like it could also be relevant, but I don't think it matches what's described here. In that case, I believe we're unnecessarily updating EndpointSlices if something irrelevant in the Pod changes (resource version), but I don't believe there's anything special there that would mean we weren't updating EndpointSlices until the resync period. So with all of that said, I believe this is fixed in 1.21+, but let me know if anyone's able to recreate this in those versions and/or if is actually more relevant than it appears on the surface.
Thanks for your reply I will check it.
I confirmed that the periodic updates in my cluster are all empty endpointslices. There are l lot of IPv6 endpointslice in my cluster. But it shouldn't be updated when endpointslice-controller couldn't list pods by service label selector.
/assign
I found that when endpointslice's endpoint array is empty, endpointslice-controller updates it every periodic resync, although it didn't change. I consider it a bug and would like to fork and fix it. The code below could fix the issue of updating empty endpointslices, but I'm not sure if it will introduce new issues. for i := 0; i < len(slicesToDelete); { if len(slicesToCreate) == 0 { break } sliceToDelete := slicesToDelete[i] slice := slicesToCreate[len(slicesToCreate)-1] // Only update EndpointSlices that are owned by this Service and have // the same AddressType. We need to avoid updating EndpointSlices that // are being garbage collected for an old Service with the same name. // The AddressType field is immutable. Since Services also consider // IPFamily immutable, the only case where this should matter will be // the migration from IP to IPv4 and IPv6 AddressTypes, where there's a // chance EndpointSlices with an IP AddressType would otherwise be // updated to IPv4 or IPv6 without this check.

if sliceToDelete.AddressType == slice.AddressType && ownedBy(sliceToDelete, service) { slicesToCreate = slicesToCreate[:len(slicesToCreate)-1] // We must consider the situation that an empty unchanged endpointslice no need to update. If the one to be deleted and the one to be created are the same endpointslice which is empty, skip the update step. // eg: If there are many headless svc with single stack ipv4 for example, a lot of ipv6 empty // endpointslice update could be avoid. if slice.Name != sliceToDelete.Name len(slice.Endpoints) != 0 { slice.Name = sliceToDelete.Name slicesToUpdate = append(slicesToUpdate, slice) } slicesToDelete = append(slicesToDelete[:i], slicesToDelete[i+1:]...) } else { i++ } /cc
Hey I'm not sure I completely understand this fix, but thanks for volunteering to take this on! As I recall, this specific section of the reconciler code is part of the "recycling" logic, where an EndpointSlice marked for creation reuses the name of when marked for the deletion, and the operation becomes an update. I'd appreciate a PR if you can also include a test that recreates the buggy behavior + shows that this fixes it.
Thanks for your reply. I will try more test and show this fix does work.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I understand the bug now - it looks the root issue is here: Essentially we'd consider the previous placeholder EndpointSlice no longer desired, mark it for deletion, and then that would mean no EndpointSlices would be left, triggering a new placeholder slice to be marked for creation.

The recycling logic you linked above essentially merges creates and deletes into updates, resulting in the behavior you described of empty EndpointSlices always being updated during the resync period. I think this should be relatively straightforward to fix with some changes to the code I referenced above.
/assign
GitHub didn't allow me to assign the following users: spencerhance. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: go.etcd.io/etcd/pkg/v3 v3.5.13 // indirect go.etcd.io/etcd/raft/v3 v3.5.13 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect <del> go.opentelemetry.io/otel/metric v1.28.0 // indirect </del> go.opentelemetry.io/proto/otlp v1.3.1 // indirect go.uber.org/multierr v1.11.0 // indirect golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect

Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29.

At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks:


Question: }) ginkgo.It("The containers terminated by OOM killer should have the reason set to OOMKilled", func() { <ins> cfg, configErr := getCurrentKubeletConfig(context.TODO()) framework.ExpectNoError(configErr) if utilfeature.DefaultFeatureGate.Enabled(features.NodeSwap) { // If Swap is enabled, we should test OOM with LimitedSwap. // UnlimitedSwap allows for workloads to use unbounded swap which // makes testing OOM challenging. // We are not able to change the default for these conformance tests, // so we will skip these tests if swap is enabled. if cfg.MemorySwap.SwapBehavior == "" || cfg.MemorySwap.SwapBehavior == "UnlimitedSwap" { ginkgo.Skip("OOMKiller should not run with UnlimitedSwap") } } </ins> ginkgo.By("Waiting for the pod to be failed") err := e2epod.WaitForPodTerminatedInNamespace(context.TODO(), f.ClientSet, testCase.podSpec.Name, "", f.Namespace.Name) framework.ExpectNoError(err, "Failed waiting for pod to terminate, %s/%s", f.Namespace.Name, testCase.podSpec.Name)

The swap fedora job is failing consistently for OOM. For a while. It looks like OOM is not actually killing these pods when swap is enabled. I did not see any examples of this failing without swap enabled so I think this is a real issue. There is a failure in OOMKilled as part of the serial fedora job but that is not related to this. This is part of and should be required for promotion to Beta2. /sig node
/triage accepted /priority important-longterm


Question: type azureToken struct { token       adal.Token <ins> environment string </ins> clientID    string tenantID    string apiserverID string

What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Close the queue. func (f *DeltaFIFO) Close() { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> f.closed = true f.cond.Broadcast() }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:


Question: shouldSchedule:        true, shouldContinueRunning: true, }, <ins> { predicateName: "ShouldRunDaemonPodOnUnscheduableNode", ds: &apps.DaemonSet{ Spec: apps.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel}, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: simpleDaemonSetLabel, }, Spec: resourcePodSpec("", "50M", "0.5"), }, }, }, nodeUnschedulable:     true, wantToRun:             true, shouldSchedule:        true, shouldContinueRunning: true, }, </ins> } for i, c := range cases {

DaemonSet should not respect node's field. It's because cluster operators often want system components to be running and ready before workloads run on a given node. OTOH, DaemonSet does respect NoSchedule taints. This behavior is broken by , because DaemonSet share some predicates code with default scheduler.
/assign
Added this to 1.10 milestone, because it's a regression introduced in 1.10.
can you (or a SIG lead) add status/approved-for-milestone please? I'll add the priority. /priority important-soon
how is this use case satisfied by the daemonset controller creating pods and depending on the scheduler to assign them to nodes? having the scheduler ignore the unschedulable field seems nonsensical
I'd like get this as limitation for alpha feature; when upgrade to beta, I'd like to use taint to replace un-scheduable, so DaemonSet pod can tolerate it on-demand.
Escalating to critical-urgent, since we are now in Code Freeze. Please update daily with status, thanks! /priority critical-urgent /remove-priority important-soon
In short term, scheduler can check of pods and ignore bit of nodes for DS pods. Long term, when TaintNodesByCondition is stable, we should replace bit by taint and have DS controller add toleration for the taint when creating pods.
That doesn't seem proper. Making pods behave differently depending on the type of their creating object is very counterintuitive. That seems more reasonable. I'm less certain about auto- tolerations
- indeed there's some work currently to move from to a taint based behaviour, tracked by


Question: "Comma-separated list of apimachinery import paths which are considered, after tag-specified peers, for conversions. Only change these if you have very good reasons.") pflag.CommandLine.StringSliceVar(&ca.ExtraPeerDirs, "extra-peer-dirs", ca.ExtraPeerDirs, "Application specific comma-separated list of import paths which are considered, after tag-specified peers and base-peer-dirs, for conversions.") <ins> pflag.CommandLine.StringSliceVar(&ca.ExtraDirs, "extra-dirs", ca.ExtraDirs, "Application specific comma-separated list of import paths which are loaded and considered for callable conversions, but are not considered peers for conversion.") </ins> pflag.CommandLine.BoolVar(&ca.SkipUnsafe, "skip-unsafe", ca.SkipUnsafe, "If true, will not generate code using unsafe pointer conversions; resulting code may be slower.") }

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: return nil } <del> return r.alloc.Release(offset) </del> <ins> err := r.alloc.Release(offset) if err == nil { // update metrics label := r.CIDR() clusterIPAllocated.WithLabelValues(label.String()).Set(float64(r.Used())) clusterIPAvailable.WithLabelValues(label.String()).Set(float64(r.Free())) } return err </ins> } // ForEach calls the provided function for each allocated IP.

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --Add metrics to apiserver tracking the service IP allocator - around how many IP addresses are used(allocated) vs available, and/or if there were any errors allocating service IPs to a service. We want to track if we are approaching the limit of the ServiceClusterIPRange, which is a fixed CIDR and difficult to change. Currently while we can approximate the usage looking at services in the cluster, it is not coming from the source allocator which is within the apiserver. Similar metrics are currently available for the node's ipam allocatior: /area apiserver /sig instrumentation /sig network
/triage accepted /assign I see something like this, gauges for and can you expand on this? a counter incremented each time an IP allocation fails?
that is one option. for contrast the nodeipam metric has an 'allocationtriesper_request' metric - i was imagining something similar to an 'attempts', but with a success:true/false tag/facet so we could compare how often allocations are attempted/how often they fail. I will defer to the experts here as I'm not sure what the status quo for reporting attempts/failures are. I'm not sure if there's any code paths that would cause allocation failures other than usage total (or available <= 0) - but having them separated out like that would be handy in that scenario. (maybe etcd internal allocation is corrupt? or context deadline exceeded)


Question: if err != nil { return false, err } <ins> available, err := c.checkIfAvailable(awsDisk, "deleting", "") if !available { return false, err } </ins> return awsDisk.deleteVolume() } <ins> func (c *Cloud) checkIfAvailable(disk *awsDisk, opName string, instance string) (bool, error) { info, err := disk.describeVolume() if err != nil { glog.Errorf("Error describing volume %q: %q", disk.awsID, err) // if for some reason we can not describe volume we will return error return false, err } volumeState := aws.StringValue(info.State) opError := fmt.Sprintf("Error %s EBS volume %q", opName, disk.awsID) if len(instance) != 0 { opError = fmt.Sprintf("%q to instance %q", opError, instance) } // Only available volumes can be attached or deleted if volumeState != "available" { // Volume is attached somewhere else and we can not attach it here if len(info.Attachments) > 0 { attachment := info.Attachments[0] attachErr := fmt.Errorf("%s since volume is currently attached to %q", opError, aws.StringValue(attachment.InstanceId)) glog.Error(attachErr) return false, attachErr } attachErr := fmt.Errorf("%s since volume is in %q state", opError, volumeState) glog.Error(attachErr) return false, attachErr } return true, nil } </ins> func (c *Cloud) GetLabelsForVolume(pv *v1.PersistentVolume) (map[string]string, error) { // Ignore any volumes that are being provisioned if pv.Spec.AWSElasticBlockStore.VolumeID == volume.ProvisionedVolumeName {

On AWS, mutable API calls such as AttachVolume or DeleteVolume have separate API quota from API calls that do not modify the resource. Currently k8s makes a high number of AttachVolume and DeleteVolume API calls even though k8s can check if the operation is going to succeed.
/sig aws


Question: err error } <del> // newGOAWAYClient return a configured http client which used to request test GOAWAY server, a dial may specified // to encounter the TCP connection. func newGOAWAYClient(dial func(network, addr string, cfg *tls.Config) (net.Conn, error)) (*http.Client, error) { tlsConfig := &tls.Config{ InsecureSkipVerify: true, NextProtos:         []string{http2.NextProtoTLS}, } tr := &http.Transport{ Proxy:               http.ProxyFromEnvironment, TLSHandshakeTimeout: 10 * time.Second, TLSClientConfig:     tlsConfig, MaxIdleConnsPerHost: 25, DialTLSContext: func(ctx context.Context, network, addr string) (net.Conn, error) { if dial != nil { return dial(network, addr, tlsConfig) } return tls.Dial(network, addr, tlsConfig) }, } if err := http2.ConfigureTransport(tr); err != nil { return nil, err } client := &http.Client{ Transport: tr, } return client, nil } </del> // requestGOAWAYServer request test GOAWAY server using specified method and data according to the given url. // A non-nil channel will be returned if the request is watch, and a watchResponse can be got from the channel when watch done. func requestGOAWAYServer(client *http.Client, serverBaseURL, url string) (<-chan watchResponse, error) {

Before merged, TestClientReceivedGOAWAY failed with data race on . However, as far as I know, it's not the root cause of TestClientReceivedGOAWAY flake, because dial a new connection only when client send a new request, and in this case, requests are sent one by one. In other words, if data race on , then the additional unexpected connection set up, which means the test case or the configuration has problem. After merged, TestClientReceivedGOAWAY still failed in many jobs, such as , , which may be a good proof of my suspect. And I got a failed job with data race issue that read at (main testing goroutine) and write at , which told us the http client try to set-up a new connection even when we had already finished all requests(may caused by goroutine leak/block in http client). I'm still debug the root cause of TestClientReceivedGOAWAY flake, for now, I doubt that I used an incorrect http2 server/client configuration( or SETTINGSMAXCONCURRENT_STREAMS limited), or the dial function ignored context.Context which will cause more than one goroutine dial the connection when the first canceled(not found any where will cancel the requests). I still can not reproduce this failed case and it's hard to find the root cause. Couple of days may be needed to find the root cause, and I will post a PR to skip TestClientReceivedGOAWAY before I solved the problem. Sorry for any inconvenient.
/assign /sig testing
cc
Thank you for working on this!! How often does it flake? I'm just wondering if we should disable it in the mean time to avoid angry contributors.
On my PC, about 1/1000. I will post a PR to disable it for now.
Thank you so much! That's not a bad rate at all, but we run a lot of tests. This is super helpful!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale I'm still working on it, a new golang version(1.16 or 1.15.7+) release required
/sig api-machinery /remove-sig testing AFAICT this is specific to these tests, not the test framework or CI infrastructure.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
the tests re-enabled in started flaking again immediately Example failure: /milestone v1.21 the tests should be deflaked or disabled for 1.21
/cc
/triage accepted
+1 in recent PR tests
Sorry for my slow response. I'm working on it, looks like I need to debug golang HTTP client again :-P
Shall we revert in the meantime?
No, has already fixed by golang 1.15.8 I think. This one is caused by another problem.
I will disable TestClientReceivedGOAWAY case again if I can not find the root cause this night.
Can not reproduce on my PC, disable this test case before I figure out the root cause:
I run this test case with this command and can not reproduce. Does any thing different with command ?
moving out of the milestone and lowering priority becuase merged, leaving open because the fundamental issue is unresolved /milestone clear
will do something comparable under make. There's a bunch of stuff in there, generally it's not relevant for a unit test bug like this but the answer to what exactly it does is not easily concise. Note that CI may have more load than your local machine running it once, and also currently runs as root.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle active /assign ref:


Question: usage := "of resource(s) to be stopped." cmdutil.AddFilenameOptionFlags(cmd, options, usage) cmd.Flags().StringP("selector", "l", "", "Selector (label query) to filter on.") <del> cmd.Flags().Bool("all", false, "[-all] to select all the specified resources.") </del> <ins> cmd.Flags().Bool("all", false, "select all resources in the namespace of the specified resource types.") </ins> cmd.Flags().Bool("ignore-not-found", false, "Treat "resource not found" as a successful stop.") cmd.Flags().Int("grace-period", -1, "Period of time in seconds given to the resource to terminate gracefully. Ignored if negative.") cmd.Flags().Duration("timeout", 0, "The length of time to wait before giving up on a delete, zero means determine a timeout from the size of the object")

It actually doesn't support --all.
/assign
ohh, thanks, will make a pr to address that right now..


Question: } func TestGetHistogramVecFromGatherer(t *testing.T) { <del> var registerMetrics sync.Once </del> tests := []struct { name    string lvMap   map[string]string

TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in


Question: framework.ExpectNoError(err, "while deleting") ginkgo.By("Checking CSI driver logs") <del> err = checkPodLogs(ctx, m.driver.GetCalls, pod, false, false, false, test.deployCSIDriverObject && csiServiceAccountTokenEnabled, numNodePublishVolume) </del> <ins> err = checkNodePublishVolume(ctx, m.driver.GetCalls, pod, false, false, false, test.deployCSIDriverObject && csiServiceAccountTokenEnabled) </ins> framework.ExpectNoError(err) }) }

1 ci-kubernetes-e2e-gci-gce 1 ci-cos-containerd-e2e-cos-gce in the master blocking test grid 1 ci-kubernetes-e2e-gci-gce-network-proxy-grpc 1 ci-kubernetes-e2e-gci-gce-proto Kubernetes e2e suite.[It] [sig-storage] CSI Mock workload info CSI PodInfoOnMount Update should be passed when update from false to true 12/14 code thaw after 1.29.0 release was merged at the time and is related to PodInfoOnMount. /cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
testCase: podInfoOnMount should be passed when update from false to true kubelet log: csi-mockplugin-0-mock log: the function in the test case is not working as expected. expect log at Line 2, but actual log is at Line 3.
is called twice for the same volume sometime even though the volume is already published successfully. Not only this issue, but also other issues like I wonder if it is a potential bug or a normal behavior. cc what do you think?
good catch. I think that any test that depends on exact index of NodePublish call will be flaky, sometimes Kubelet has to retry NodePublish. It would be better to check for NodePublish parameter changes. For example, test should check there were 1 or more NodePublish calls with no pod info and then 1 or more NodePublish calls with pod info afterwards.


Question: return len(podSpec.SchedulingGates) != 0 } <del> // SeccompAnnotationForField takes a pod seccomp profile field and returns the // converted annotation value func SeccompAnnotationForField(field *api.SeccompProfile) string { // If only seccomp fields are specified, add the corresponding annotations. // This ensures that the fields are enforced even if the node version // trails the API version switch field.Type { case api.SeccompProfileTypeUnconfined: return v1.SeccompProfileNameUnconfined case api.SeccompProfileTypeRuntimeDefault: return v1.SeccompProfileRuntimeDefault case api.SeccompProfileTypeLocalhost: if field.LocalhostProfile != nil { return v1.SeccompLocalhostProfileNamePrefix + *field.LocalhostProfile } } // we can only reach this code path if the LocalhostProfile is nil but the // provided field type is SeccompProfileTypeLocalhost or if an unrecognized // type is specified return "" } </del> func hasInvalidLabelValueInAffinitySelector(spec *api.PodSpec) bool { if spec.Affinity != nil { if spec.Affinity.PodAffinity != nil {

Graduate seccomp to GA, as defined on the respective . This is an umbrella issue to track all outstanding work. [x] API Server changes (MERGED) [x] Kubelet changes (MERGED) [x] Mark flag as deprecated (MERGED) [x] Update PSP to support seccomp fields (MERGED) [x] Document Kubernetes seccomp support (MERGED)[x] Add seccomp least privilege for docker sandbox (MERGED) [x] Add seccomp least privilege for kuberuntime (MERGED) ) /area kubelet /area kubelet-api /sig node /area security /assign
/cc
just a heads up that we have concluded all the core items for seccomp to go GA. There are still 2 pending PRs that ideally would also go in 1.19, but we are waiting for them to be reviewed: and
needs to be rebased apparently and isn't just waiting in the merge pool as verify is not passing.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think the warning mechanism was blocked by a not existing API. This is now tracked in
/remove-lifecycle stale
Updated the description to given that the warning mechanism is now set to that milestone.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/priority backlog
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen What is the remaining work here?
I guess we have to follow-up on and implement: I can take care of that if we agree that we want to ship that in 1.27. /cc


Question: // Check all affinity terms. for _, term := range GetPodAffinityTerms(affinity.PodAffinity) { <del> termMatches, matchingPodExists, err := c.anyPodMatchesPodAffinityTerm(pod, filteredPods, node, &term) </del> <ins> termMatches, matchingPodExists, err := c.anyPodMatchesPodAffinityTerm(pod, filteredPods, nodeInfo, &term) </ins> if err != nil { errMessage := fmt.Sprintf("Cannot schedule pod %+v onto node %v, because of PodAffinityTerm %v, err: %v", podName(pod), node.Name, term, err) glog.Error(errMessage)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --It is not a secret that . In my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. However, in most practical scenarios users do not need all the flexibility. For example, people often do not use arbitrary labels in topology key of the affinity rules. They often use node, zone, or region as the topology. Node in particular is used often. That's why I am proposing a fast path for the case that topology key is Node. Today, the affinity/anti-affinity predicate does not have any distinction for topology keys and in the cluster regardless. The logic should recognize "" and scan only pods on the same node for this topology key. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature /sig scheduling cc/
ref/
/assign
GitHub didn't allow me to assign the following users: misterikkit. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
We found some other simple optimizations that had a significant impact on the affinity predicate. I will upload PRs for those as well.
Reopening this as the performance of anti-affinity still doesn't seem to be satisfactory enough on large clusters (i.e 5k nodes) - and it's causing them to fail to startup. Ref: issue I can help with evaluating any performance improvements if needed.
The change sent improved performance of the predicate, but apparently it is not enough to make our large scale scalability tests happy.
This issue is not about general performance improvment of affinity/anti-affinity. It was to implement a particular improvment which is already implemented. There are other performance improvement issues. I am going to close this one.
Sounds good. Thanks.
How come with kops 1.10.0 I still see all kube-dns pods on the same node because the kube-dns deployment doesn't specify any anti-affinity rule?
I think you are correct. you can double-check with the option to . I have a v1.9 cluster in front of me, and kube-dns has no affinity rules. (I didn't use kops though)


Question: "strconv" "strings" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" unversionedvalidation "k8s.io/apimachinery/pkg/apis/meta/v1/validation" "k8s.io/apimachinery/pkg/labels"

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: DigitalOcean OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others: What happened: when I do: it will run rolling update ONLY if there is a change. In case I do: Each time I run it - it will execute rolling update, which is misleading. What you expected to happen: I would like to have consistent behavior for Deployment and DaemonSet How to reproduce it (as minimally and precisely as possible): Create any DaemonSet with following:
Hmm, it shouldn't happen. TemplateGeneration is updated only when something changed. I will debug it.
I can reproduce this, and the DaemonSet is updated after every , even when nothing is changed. In , old/new DaemonSet template are seen as different with . I tried to change this part to use instead (needed to update code in validation as well), and the DaemonSet templates were then seen as the same, and thus the issue was gone. We used everywhere in the controllers. Is the right way to compare stuffs?
It's very strange. I tried to use pretty.Diff to print the differences but it returns empty slice. apiequality.Semantic.DeepEqual uses forked version of reflect.DeepEqual and it's little different from version in 1.7. will you prepare the patch?
The key of the reported DaemonSet config is its . Note that in the config file, (empty list); however, in the live DaemonSet object (created from the config file), (null list). That's why thinks they're different. If we use in the config file instead, won't change. From the field description, empty list and null list are different. Feel it's api machinery related
protobuf storage isn't able to store , so the object has that field set to when read back from storage. that is causing problems for that field in particular (where and have different meanings) see
cc for behavior of namespaces field
so it's not a DaemonSet bug
Well, still not sure whether Semantic.DeepEqual should be used or not. I'm still concerned with the behavior of the controller for other fields that can be populated with non-null empty lists by users, but serialize back out to null (or set omitempty)
deployments are using Maybe we should also switch?

If empty lists should be treated differently from null list, we should replace with in controller code.
that distinction is going away in all the fields I know of... I'm not super happy with Semantic.DeepEqual masking the difference between those two things, but that's probably the right thing for you to be using
Special meaning of [] was removed in I think a switch to Semantic.DeepEqual is still needed here
Filed


Question: --network="${KUBE_GCE_NETWORK:-${KUBE_GKE_NETWORK:-e2e}}"  --node-tag="${NODE_TAG:-}"  --master-tag="${MASTER_TAG:-}"  <ins> --docker-config-file="${DOCKER_CONFIG_FILE:-}"  </ins> --dns-domain="${KUBE_DNS_DOMAIN:-cluster.local}"  --ginkgo.slowSpecThreshold="${GINKGO_SLOW_SPEC_THRESHOLD:-300}"  ${KUBE_CONTAINER_RUNTIME:+"--container-runtime=${KUBE_CONTAINER_RUNTIME}"}

"[] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance]" is failing consistently for Windows 1909 on GCE, but it's passing for 2019. Likely one of the containers it depends on hasn't been rebuilt as multi-arch? Failure message from
/sig windows cc
More useful output from build- Seems the container doesn't work on 1909. cc
/sig windows
This test is also failing on for Windows 1903.
see
Ah, thanks, I searched k/k for an existing issue but didn't think to search windows-testing. I posted a follow-up there.
Hi. It is a known issue, I've also reported it a while back:
, do you have push right to the ` manifest list to include the newer Windows OS versions. Basically, the commands are: docker manifest create docker manifest push --purge
/kind failing-test
I tried running this but got: I looked into the project and noticed and as owners. Are you able to assist?
/assign
docker manifest push Pushed ref with digest: sha256:ec76696da5a660a02bbfd304549d6e09d8c9fe48f2d4efe642f26cb1e46a6bde Pushed ref with digest: sha256:7c4ee446a647fdf8988672ae5e685cb66d53d4581e057b02e3a2bad728177f48 Pushed ref with digest: sha256:297cce9134ddaa2c09d91dea7ab7af0e58e06e59858c0311a0770143d144ddf1 sha256:fb9b25770487567c02bf90dd3edea7917323556d1b7ba81ec042ffd5f9effeae
this is fixed. the image manifest is available in
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: containerd-l2bridge-windows-master containerd-overlay-windows-master Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] Since when has it been failing: Unknown. Testgrid link: Reason for failure: When creating the container, the following error occurs: Failed: Error: failed to create containerd task: hcsshim::CreateComputeSystem image-pull-test: The container operating system does not match the host operating system. Windows Server 2019 is used in the testing environment, and the image image, and the test is passing in all other jobs that are using docker instead of containerd. But the image has been built at the beggining of 2019, so it's most likely outdated (the Microsoft official images are updated regularly), which could lead to the OS version mismatch. Note that most of the other tests are passing for the containerd jobs, which are using the test images hosted in the image (LTS 2019 and 1809 have the same OS version), and they're always up-to-date. One solution could be to republish the image based on a freshly pulled nanoserver image. Anything else we need to know: /sig windows /sig testing
- is the login/password public so this can be debugged outside of the prow cluster?
- what containerd branch / hash was it built from?
- Some part of my memory thinks this was fixed in upstream containerd/cri. Do you recall any items around this recently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: # See the OWNERS docs at https://go.k8s.io/owners <ins> # Disable inheritance as this is an api owners file options: no_parent_owners: true </ins> filters: ".*": approvers:

I believe this owners file should not inherit permissions from parent OWNERS files; only api approvers should be able to approve api changes.
/kind feature /sig architecture
opened to do this


Question: panic(fmt.Errorf("invalid character in component [%q] of CgroupName", component)) } } <del> return CgroupName(append(base, components...)) </del> <ins> // copy data from the base cgroup to eliminate cases where CgroupNames share underlying slices.  See #68416 baseCopy := make([]string, len(base)) copy(baseCopy, base) return CgroupName(append(baseCopy, components...)) </ins> } func escapeSystemdCgroupName(part string) string {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Setting a Resource requests and limits on a Pod other Pods running on the node where it was scheduled are moved to the cgroups created to serve the QoS request. What you expected to happen: Pods to stay in their assigned cgroups. How to reproduce it (as minimally and precisely as possible): kubelet running with the following flags: Create a pod without resource reservations and then create another one with resource reservations. For example, using the following template: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig node
This is the memory cgroup tree layout before starting an pod with resource request and limits. This is after the pod is created. Notice that is moved to the cgroup where nginx is running in
Interesting, since is running as a static pod
/milestone v1.12
this looks off... can you provide the output for the kube-proxy yaml file? what container runtime is being used? can you share the kubelet configuration?
We need to figure out if this is a regression in 1.12. Can you take a look?
I can reproduce it on GCE:
Node with same flags, except , instead of `--cgroup-root="/dashpole/" So the problem only occurs when is specified
I can reproduce this on 1.11.3 as well. I don't think this is a regression with the 1.12 release, although it is still a serious bug.
Correct, was being specified in the case above. As mentioned over Slack, we were able to track it back to 1.11.0-beta (can't recall the beta version, though) but not 1.11.0-alpha.2. The only PR we've identified which could have an impact is .
I will try to take a look today. I am curious the motivation for changing the cgroup-root, in retrospect, would have preferred we didnt expose it.
I am unable to reproduce this using systemd cgroup driver when using Given this exists currently in 1.11, we should fix this, but it should not block 1.12 milestone.
Thanks /priority important-soon
can you help to reproduce?

is OOO until Monday.
i just ran out of time to validate this with alternate driver (had assumed it would have happened with both), but it appears localized to one.
Nevertheless it's a good datapoint, it means it's only affecting cgroupfs driver. Regarding your question, the cgroup-root flag was the cleanest way we could find to have the kubelet respect the Mesos cgroup setup. Lmk if I can help you with anything.
ping , is this on track for 1.13? any related PRs?
I haven't looked deeper into this yet. Ill keep this on my list for 1.13
any porgress on this ? I don't see any associated PRs linked here. We are nearing code slush for 1.13 next Friday 11/9 and Code freeze in 2 weeks. If you think this issue isn't critical for 1.13 and work might extend beyond code slush please work with Niko Pen to move this out. thanks
Ill pick up this work over the next few days
I have narrowed it down to this strange behavior in golang: Basically, appending to a list constructed with has different behavior than appending to a list constructed literally, but only with lists where the size is larger than 1... This strange behavior with golang slices results in the "best-effort" qos cgroup in the qosContainerManager being changed to a pod's cgroup.
A simpler reproduction case:


Question: Expect(err).NotTo(HaveOccurred(), "failed to fetch pod: %s in namespace: %s", execPodName, ns) if !isTransitionTest { <del> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true, false)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true)).To(BeTrue()) </ins> } if isTransitionTest { svc = jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.SessionAffinity = v1.ServiceAffinityNone }) <del> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, false, true)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, false)).To(BeTrue()) </ins> svc = jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.SessionAffinity = v1.ServiceAffinityClientIP }) <del> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true, true)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true)).To(BeTrue()) </ins> } }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-master-slow Which test(s) are failing: [sig-network] Services should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] Since when has it been failing: This test started to be flaky on 11/18 Testgrid link: https://k8s- Reason for failure: Looks to be a pretty straightfowards test not passing: If we look at triage, this test has always been flaky, but it got significantly more flaky (2X failures) on 10/27: Anything else we need to know: This job is non-blocking for 1.13, except if this represents an actual bug in Affinity the 1.13 team needs to know ASAP. This failure may or may not be related to . It looks like this test was disabled for large clusters, but maybe the problem wasn't the cluster size. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig network /priority important-soon attn:
FWIW this seems to be a long standing issue, atleast as far bck as 1.12 timeframe
Sure. But the failures have become 3X as frequent, so I still want a sign-off by sig-network.
cc
Ack. Will work on this.
we plan to lift code freeze for 1.13 tomorrow,11/28 5pm PST. If there is a new regression in 1.13 thats increased the flakes and needs addressing we need to get the fix in asap. Can you please let us know if this is a blocker for 1.13 by tomorrow, 11/28, morning 10am PST ?
Great observation. I checked on the commits around that date but wasn't able to find any change that may cause regression. Hasn't been able to reproduce the same failure locally as well.
I just noticed that the same test is flaking a lot more on 1.12 branches as well https://k8s-
Good catch on that, so the recent increase of flakiness likely is not caused by 1.13 changes so we shouldn't block release on this either. cc
But we will be looking deeper into the root cause.
Several observations: It only happens when is false.

When it's true, the test does more stuff before actually check service session affinity. It doesn't happen when executing in pod, probably due to the same reason above, the test start test pod before actually testing service. I wrote a script which does the same thing in my cluster of version 1.11.3-gke.18 in us-central1-a. cannot reproduce it. The hypothesis is that the service session affinity spec takes some time to reconcile after the service is created. It could happen either on the node or on the GCLB. Since I haven't been able to reproduce it's hard to tell. But it's more likely on GCLB side, even it could be regional issue. Looks like a sleep of seconds will deflake it. but it's just hiding issues. I would rather open another issue to track the real issue.
The most recent failure is here: Basically what happened is GCLB doesn't honor session affinity and still sends the request to different VMs. The log shows that 3 requests are routed to 3 different VMs. Dec 19 07:40:58.094: INFO: [service-fsn65 service-bf8nc] Dec 19 07:40:58.094: INFO: Affinity should hold but didn't. Dec 19 07:40:58.196: INFO: [pod,node] pairs: [{Pod:service-bf8nc Node:bootstrap-e2e-minion-group-f3nb} {Pod:service-fsn65 Node:bootstrap-e2e-minion-group-9sp7} {Pod:service-rnrcw Node:bootstrap-e2e-minion-group-m5rj}]; err: <nil
More recent failures: Logs:
Hello, could this issue be prioritized to help with stabilizing the release blocking upgrade tests? They're currently flaking ~23% of the time in () ~30% of the time in ()
/assign can you take a look
GitHub didn't allow me to assign the following users: take, a, look, can, you. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.

For more information please see <detailsIn response to : /assign can you take a look Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am preparing a PR, basically allow Affinity some time to be configured, instead of failing out instantly.
Thanks! Will keep an eye out for it.
The test is no longer flaky: https://k8s- Thanks everyone!
Thanks! Will keep an eye on it for a few days, on the following dashboards:
It's stabler now, but still flaking once/twice a day, e.g. in . - Is this the same issue as you looked at? Is there more to be done here? /reopen
Reopened this issue. <detailsIn response to : It's stabler now, but still flaking once/twice a day, e.g. in . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
we should probably cherrypick into 1.13 in case of deflaking upgrade/downgrade job, because those jobs would run 1.13 tests against 1.14 (upgraded) cluster or 1.13 (downgraded) cluster. FYI we've hit similar situation for DNS test (ref ).
Good point. Cherrypick PR prepared.
14: <img width="1110" alt="Screen Shot 2019-03-16 at 12 11 07 PM" src="https://user-"13: <img width="1091" alt="Screen Shot 2019-03-16 at 12 11 22 PM" src="https://user-"Do we want to cherry-pick this back further?
Thanks for the nice graph. Look like the fix works. I'll create cherrypick PRs for 1.11 and 1.12 right away.
AFAICT this has been significantly improved: ! Closing this and thanks to /close
Closing this issue. <detailsIn response to : AFAICT this has been significantly improved: ! /close Instructions for interacting with me using PR comments are available .


Question: return nil, err } <ins> if config.VMType == "" { // default to standard vmType if not set. config.VMType = vmTypeStandard } </ins> env, err := auth.ParseAzureEnvironment(config.Cloud) if err != nil { return nil, err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When using IMDS, kubelet reports even when vmtype is not set: What you expected to happen: azure cloud provider should assume stardard vmtype if it is not set. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
Added to v1.10 milestone since it is a critical bug fix. cc


Question: "target": "//plugin/cmd/kube-scheduler:kube-scheduler", }, "kube-proxy": { <del> "base": ":busybox-net", </del> <ins> "base": "@debian-iptables-amd64//image:image.tar", </ins> "target": "//cmd/kube-proxy:kube-proxy", }, }

BUG REPORT Kubernetes version 2 Environment: Install tools: What happened: The code in takes care to supply the flag so it will wait on a lock to avoid interference between multiple processes modifying the table. If you read the source you discover it does this by creating a file . However, when is running in a container, is in a local filesystem, so it is not able to lock against processes running on the host or in other containers. And before you decide that the answer is to mount from the host, look at all the issues around leaking netns, e.g. Credit for pointing this out to me. Credit for pointing out locking functionality was to a few months back
Ironically, this is caused by a bugfix that moved to file-based locking. Previously, iptables would lock by reserving an abstract unix socket, which is tied to the network namespace.
Some discussion also in and following comments.
I experimented with this: but it didn't work on CoreOS, as the host wasn't using a version of iptables that used the file lock yet. Just ended up creating a directory call
I guess I should file this as an issue against kubeadm and any of the other sources of kube-proxy container config.
/assign
Should we leave this open (waiting for ... ?)
Waiting for I guess... (which I'm reviewing now)
i think this should not be marked as complete until the kubeadm fixes are in.
/reopen
/close fix in


Question: expectedRefs int }{ { <del> "/dev/disk/by-path/prefix", </del> <ins> "/127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00", </ins> 4, }, }

During iscsi volume cleanup, kubelet reads from and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of . But currently the device path in is in the form of , thus iscsi panicked . I suspect this panic started to happen after switched to use from . seems to always resolve device path (see output below), and thus the expected iscsi configuration in the device path no longer exists.