Question: <del> v1.12.4-1 </del> <ins> v1.12.5-1 </ins>

Something has changed between 320 and 321 runs and the request latency has increased significantly. The graphs below shows request latency (non watch) over time for a single test run: : ! : ! In my opinion they looks significantly different: in 320 most of the spikes are lower than 0.8s while 321 most of the spikes exceeds 1.6s. I checked few runs below 320 and have latency pattern similar to 320 and few runs after 321 and they have similar pattern to the 321. We are seeing similar latency growth in kubemark tests. The most significant change between 321 and 320 was go 1.11.5 -1.12 migration. /cc
/sig scalability
With we are trying to prepare small scale repro that we can use for further debugging.
was also in that range (cc for verification that nothing in that PR increased server load in non-alpha clusters)
did tests at head with reverting to go1.11 and metrics seemed better He is working on confirming that and should paste results then, but it seems to be pretty clear that it's go regression. We're also working on providing smaller-scale repro case to work closer with golang team (they used to be really helpful when debugging regressions in previous versions (IIRC 1.9 is where there were previous one).
I double checked and didn't find anything suspicious. The alpha function wasn't invoked in that window, and the PR is a no-op to non-alpha clusters
Regarding what wrote: ! I was mostly investigating the regression on kubemark-5000. Some of its latencies increased x3. I run two tests: one with checkout, and the other one with some commit checkout but without (Update to use golang 1.12). The 99th latency percentile of node listing was 3400.152ms in first case and f 952.576ms in second one.
Look into other metrics (I can't remember which exactly, delete pods? post pods? or sth like that) - which increased by almost 10x.
This is the graph that I had on my mind, but yeah, it's more like 4x not 10x. But still quite huge drop: !

Reproducing the issue: kubernetes repo - First broken version (go1.12) Last working version (go1.11) perf-tests repo - This repo is home for clusterloader2 - tool used for running the scalability tests. kubernetes directory run “make quick-release”. This will build kubernetes based on checkout version. GCE cluster. kubemark test. This step creates kubemark cluster (on top of the cluster created in step 4), run cluster loader test and tear down the cluster. The test run metrics will be exported to directory provided by the --report-dir flag. Inside this directory there will be APIResponsivenessload file containing 50, 90, and 90 percentiles of all apiserver call latencies. The latencies affected the most are: Listing nodes { "Resource": "nodes", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } Pod deletion { "Resource": "pods", "Scope": "namespace", "Subresource": "", "Verb": "DELETE" } Listing services { "Resource": "services", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } I would suggest their 99th percentiles as the issue metric. As for comparison: -Go1.11 run: Listing nodes - 162.409ms Pod deletion - 41.302ms Listing services - 28.277ms -Go1.12 run: Listing nodes - 293.724ms Pod deletion - 56.2ms Listing services - 46.183ms down GCE cluster
We're trying to dig deeper into it to both root cause it to a smaller part of the system, but it is quite tricky. Unfortunately, we don't have much more than guesses right now, though. Some observations: I looked into kubemark-5000 runs into DELETE pods calls, and it seems that all of them for which we have traces (500ms+) - there are ~5000 of those in each of recent runs [previously 99th percentile was <200ms), there is a significant big of "Object deleted from database". That may suggest some etcd issues, but we don't see logs related to long-running-calls on etcd side.

That may suggest some issues on the etcd client side, but we don't have anything concrete yet.It's also possible that golang upgrade changes something in serialization/deserialization - that's a second hypothesis (we've seen a bunch of traces of "Decoded delete options" (deserialization is also done in etcd client)), but I'm not convinced this is the case as of now
We've spent significant time with golang folks and already have some findings. To summarize: the microbenchmark from enabled Go team to find some regression on their side - the faulty commit from golang is: to cite Go folks: "To summarize the change: I removed the "fast" path for small page-level allocations (which currently always need to acquire the heap lock) in order to simplify code for subsequent changes. In effect, span allocations now all go through a balanced tree rather than just a linked list for small spans (I think it was all spans under 128 pages?)." We aren't fully convinced that this is for sure the same thing that is causing regression in our scalability tests, but analyzing traces, profiles, golang tracesetc. suggests that problems are indeed around memory allocations or GC. So with high probabilty this is the same thing. We are discussing with the way to prove it's the first thing and working with the to see what we can do with it.
It appeared that the commit isn't the most faulty one for us - running scalability tests against that commit doesn't show significant drop in metrics. In the lack of better ideas, we decided to bisect changes in golang by running full scalability suite on 5k-node cluster against them. This will take couple more days though...
Just to update this thread - we're bisecting Go 1.12 branch and there aren't that many remaining commits. We already know that: is a good one is a bad one I hope that we will know the exact faulty commit on Monday.
Looking through the following stand out for me /me places bets on one of those 2 :)
That is our bet too - we're working with Go folks to confirm and fix/mitigate this.
Any luck? (curious!)

I'm sorry for delay with update here - the current update is as following: after the next round of bisections, we know that: is a good one is a bad one During my discussions with golang folks, the outcome was that we believe that is the faulty one, as it adds a bunch of syscall to second-slowest allocation path (built on top of the previous one) should only help So the next step was to verify and indeed it is a bad one (and actually as expected, the metrics were worse than in the so far confirmed oldest bad commit: ) This is what we already know. What we're doing now is that golang folks provided us two patches that they think should mitigate the issue and I'm trying to confirm if that is really true. We should have results of this one later today. The two potential outcomes are: it solves our issue; then I will be working with golang folks to patch 1.12 with those it doesn't solve our issue: then I will be working on final confirmation that it's exactly this commit (by checking the previous one, and the next one to confirm it really helps) and working further with golang folks on what else we can do to mitigate it (and also I will try to understand deeper what exactly is happening there, I just have some rough understanding now).
Very very cool! thanks
cc
Just to give an update on that.

we fully confirmed that the regression is due to (using go1.12.1 with manually reverted that commit reverts the behavior) however, according to golang fokls that commit was fixing some things, and there are couple commits built on top of them after 1.12, so it's not-trivial to revert it (we are still looking for other options)the patches that golang team provided helps, but don't fix the problem (they roughly reduce the problem to 50%) yesterday I got another two patches that may help and we're validating them nowin the meantime we're also working with golang folks to try gathering more useful data for them for debugging - they already have some patches to apply in golang and I'm working on piping them through k8s and our build scripts - I hope I will have them ready today for an over-night run
To give an update on that - the current state of it is as following: that particular commit can't be just reverted, as it was fixing some issue however, golang folks have a PR that is mitigating that issue in a different way we have verified with custom build of that fully mitigates regression on our side The current plan is as following: has been merged to head of Go [that is the mitigation commit] it will be cherrypicked to 1.12 branch additionally in 1.12 we will revert the offending commit: We expect a new patch release of Go with those two commits to be released on May 1st. is currently my main point-of-contact coordinating that on golang side.
thanks for the update ! we will still be in 1.15 alpha* phases so that's great news.
- the trick is that go 1.12 is also in 1.14, so we should also cherrypick 1.14 branch - currently there is visible regression in 1.14
agree. i'd support that
The commits that we were waiting for are already there: Now we are waiting for next 1.12 patch release of Go to update Kubernetes to use, the ETA is May 1st. - given that I will be on vacation for the rest of this week, would you be able to help with that?
yes i can, please let me know what needs to be done (so i can figure out if i have access before you head out)
- thanks a lot.

I didn't do upgrade of go for 2 years or so, so I'm not entirely sure how (if?) the process changed in the meantime. - was doing the last one: , so I'm sure he will be able to help with that.
Ah just update to latest golang when it gets out. yes, i can do that. i was wondering if you wanted me to kick off some CI runs or something. Have a good vacation
- thanks a lot! Yes - it wasn't clear, but I was testing that and it helps us, so no need to run another tests (we will get final confirmation from postsubmits).
I can also help out to get this upgrade in when it’s released. Thanks for all your work on this
Looks like 1.12.5 went out this afternoon. PR open (), but expect tests to fail: Waiting for the docker hub image to publish. The PR is merged, so this should happen shortly. Need a new version of the bazel rules_go to be published.
Just to clarify the above (from : "oh, so I saw the auto update ran here: but it looks like the update for the tags hasn't merged yet: "
Yea, it takes some time for the images to get published.
Looks like new images are live.
Reopening this so we can verify that everything is back to normal in CI
/assign
For me it's pretty clear that this was resolved: http://perf- http://perf- Now there is one more thing we should do, which is cherrypick it to 1.14 branch. ^^
Opened pick:
has SIG Scalability done any one-off custom scale/perf testing with the new golang patch release and the k8s 1.14 branch content? Do you see this cherry pick as a 1.14.2 release blocker? Or would a week or two of focused testing with updated golang in the 1.14 k8s branch and a 1.14.3 release by end of May be a sufficiently safe yet expedient response?
We were doing a bunch of tests to discover the real regression in golang 1.12 - the regression happened within 1.14 release timeframe and the bisection was done on that branch. So yes - I'm 100% confident that it helps.

Tough question - I would love to see it in 1.14.2, otoh given how long it took to debug and fix it, I can see release team pushing it to 1.14.3. But if we could have that in 1.14.2, it would be extremely helpful.
just merged and we let 1.14 soak for one or two days before we cut . Let's keep this issue open until we have some data on the latencies after that merge. SIG Scalability, can you please provide some data / graphs about the latency in a (couple of) days for the branch?
is there any evidence we'll be able to observe in for validation? Or is it possible for SIG Scalability to trigger any custom runs now that the cherry-pick has merged to the 1.14 branch?
We aren't running large tests on release branches due to cost of them... We don't have any free quota to run it now, due to huge number of regressions that we have at head that we're currently debugging. But I'm 100% confident with this change. What I was personally doing in the past: I was running tests against commit when we bumped to 1.12.0 with 1.12.5 (which was from beginning of March, so still in 1.14 timeline) and the effect was exactly the same as the effect against head we got after merging it. I also understand changes in Go between 1.12.4 and 1.12.5 and their effect on performance And we also understand this regression, and why the fixes help, which has nothing to do with any changes that are happening to kubernetes. So I'm entirely sure that if you didn't observe any correctness issues, from performance perspective it is good-to-go.
Thanks for your answer!
With this being fixed in head and 1.14.2 released with the fix, I'm closing this one as fixed.


Question: import ( "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/watch" </ins> ) // FakePods implements PodsInterface. Meant to be embedded into a struct to get a default

PodInterface should have Watch method behaving in the same way as ServiceInterface.Watch does. Use case: a monitoring system (namely Prometheus) expects to get the list of monitored servers performing a DNS-SD query. Special service (similar to kube2sky shipped with Kubernetes) mirrors status of running Pods into etcd, which is in turn monitored by SkyDNS server and exported as DNS-SD. Schematically the data flow looks like this: Kubernetes -kube2sky -etcd -SkyDNS -Prometheus. The biggest challenge is to deliver list of all running pods from Kubernetes to kube2sky. Currently kube2sky can only poll Kubernetes with PodInterface.List() each N seconds, which is not efficient. Having a Watch method would be much better.
cc Do you know why we do not support watch?
Every object should support Watch.
Watching status doesn't currently work -- that's .
I agree with OP. This is just an accident of history; some boilerplate in the client isn't completely filled out. As Brian mentions, not every field is currently watchable but that's being addressed elsewhere.
This is relevant to your interests potentially w/ your work on
This is a pretty simple change if someone wants to get their feet wet. :)
I'll work on it.


Question: defer ctrl.Stop() // This PV will be claimed, released, and recycled. <del> pv := createPV("fake-pv", "/tmp/foo", "10G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}, api.PersistentVolumeReclaimRecycle) pvc := createPVC("fake-pvc", "5G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}) </del> <ins> pv := createPV("fake-pv-recycler", "/tmp/foo", "10G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}, api.PersistentVolumeReclaimRecycle) pvc := createPVC("fake-pvc-recycler", "5G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}) </ins> _, err := testClient.PersistentVolumes().Create(pv) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } <ins> glog.V(2).Infof("TestPersistentVolumeRecycler pvc created") </ins> _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } <ins> glog.V(2).Infof("TestPersistentVolumeRecycler pvc created") </ins> // wait until the controller pairs the volume and claim waitForPersistentVolumePhase(testClient, pv.Name, watchPV, api.VolumeBound) <ins> glog.V(2).Infof("TestPersistentVolumeRecycler pv bound") </ins> waitForPersistentVolumeClaimPhase(testClient, pvc.Name, watchPVC, api.ClaimBound) <ins> glog.V(2).Infof("TestPersistentVolumeRecycler pvc bound") </ins> // deleting a claim releases the volume, after which it can be recycled if err := testClient.PersistentVolumeClaims(api.NamespaceDefault).Delete(pvc.Name, nil); err != nil { t.Errorf("error deleting claim %s", pvc.Name) } <ins> glog.V(2).Infof("TestPersistentVolumeRecycler pvc deleted") </ins> waitForPersistentVolumePhase(testClient, pv.Name, watchPV, api.VolumeReleased) <ins> glog.V(2).Infof("TestPersistentVolumeRecycler pv released") </ins> waitForPersistentVolumePhase(testClient, pv.Name, watchPV, api.VolumeAvailable) <ins> glog.V(2).Infof("TestPersistentVolumeRecycler pv available") } </ins> <del> // end of Recycler test. // Deleter test begins now. // tests are serial because running masters concurrently that delete keys may cause similar tests to time out </del> <ins> func TestPersistentVolumeDeleter(t *testing.T) { glog.V(2).Infof("TestPersistentVolumeDeleter started") _, s := framework.RunAMaster(t) defer s.Close() </ins> deleteAllEtcdKeys() <ins> testClient, ctrl, watchPV, watchPVC := createClients(t, s) defer watchPV.Stop() defer watchPVC.Stop() ctrl.Run() defer ctrl.Stop() </ins> <del> // change the reclamation policy of the PV for the next test pv.Spec.PersistentVolumeReclaimPolicy = api.PersistentVolumeReclaimDelete </del> <ins> // This PV will be claimed, released, and deleted. pv := createPV("fake-pv-deleter", "/tmp/foo", "10G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}, api.PersistentVolumeReclaimDelete) pvc := createPVC("fake-pvc-deleter", "5G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}) </ins> <del> _, err = testClient.PersistentVolumes().Create(pv) </del> <ins> _, err := testClient.PersistentVolumes().Create(pv) </ins> if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } <ins> glog.V(2).Infof("TestPersistentVolumeDeleter pv created") </ins> _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } <ins> glog.V(2).Infof("TestPersistentVolumeDeleter pvc created") </ins> waitForPersistentVolumePhase(testClient, pv.Name, watchPV, api.VolumeBound) <ins> glog.V(2).Infof("TestPersistentVolumeDeleter pv bound") </ins> waitForPersistentVolumeClaimPhase(testClient, pvc.Name, watchPVC, api.ClaimBound) <ins> glog.V(2).Infof("TestPersistentVolumeDeleter pvc bound") </ins> // deleting a claim releases the volume, after which it can be recycled if err := testClient.PersistentVolumeClaims(api.NamespaceDefault).Delete(pvc.Name, nil); err != nil { t.Errorf("error deleting claim %s", pvc.Name) } <ins> glog.V(2).Infof("TestPersistentVolumeDeleter pvc deleted") </ins> waitForPersistentVolumePhase(testClient, pv.Name, watchPV, api.VolumeReleased) <ins> glog.V(2).Infof("TestPersistentVolumeDeleter pv released") </ins> for { event := <-watchPV.ResultChan()

Test failed with and the relevant part of the stack trace is: Logs:
(or anyone), integration tests do not log much and I don't know what's going on in volume controller. Would it be possible to raise integration test log level at least to v=3 (4 preferred)?
what is the status of
can you please take a look?
Increasing integration log level can go into I think. It's probably a good idea to split them into their own job.


Question: # copy the binaries and scripts to the ~/kube directory on the master scp -r $SSH_OPTS  saltbase/salt/generate-cert/make-ca-cert.sh  <ins> easy-rsa.tar.gz  </ins> ubuntu/reconfDocker.sh  "${KUBE_CONFIG_FILE}"  ubuntu/util.sh

cluster/saltbase/salt/generate-cert/make-ca- sets but does not echo what it is doing, so in the case of an error there is no evidence of which command produced the error.


Question: "k8s.io/client-go/discovery" "k8s.io/client-go/dynamic" "k8s.io/client-go/informers" <ins> "k8s.io/client-go/kubernetes" </ins> "k8s.io/client-go/kubernetes/fake" restclient "k8s.io/client-go/rest" "k8s.io/client-go/util/workqueue"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector.

When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.


Question: klog.V(2).Infof("Attach operation successful: volume %q attached to node %q.", volumeSource.DataDiskURI, nodeName) } else { klog.V(2).Infof("Attach volume %q to instance %q failed with %v", volumeSource.DataDiskURI, nodeName, err) <del> return "", fmt.Errorf("Attach volume %q to instance %q failed with %v", volumeSource.DiskName, nodeName, err) </del> <ins> return "", err </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: return &Result{err: err} } } <del> return &Result{err: fmt.Errorf("resource(s) were provided, but no name, label selector, or --all flag specified")} </del> <ins> return &Result{err: fmt.Errorf("resource(s) were provided, but no name was specified")} </ins> } return &Result{err: missingResourceError} }

What happened: Kubectl patch sub-command shows wrong error help message if an input is wrong. The error message says and when I try to use flag it says . What you expected to happen: I expected either the first error message to not have or should have a flag . How to reproduce it (as minimally and precisely as possible): If you run the following command without a node name you get this error: As the above error said you can specify , you specify that. This fails with another error saying is unknown flag: And in the help there is no flag: Environment: Kubernetes version (use ):
/sig cli
I think we need to decide whether to add flag --all and --selector to kubectl patch. After the decision, i can help.
/assign
on my queue, will look tomorrow :D
/triage accepted Probably just need to update error message here.
oh sorry, missed that one. Will look now
/assign
OK, so we decided to correct the error message per the PR right? :)
as you're dealing with this PR, I'll left you as assignee and follow the PR review :) /unassign


Question: usingEviction:                   false, getPodFn:                        getPodFn, onDoneFn:                        d.OnPodDeletedOrEvicted, <ins> onFinishFn:                      d.OnPodDeletionOrEvictionFinished, </ins> globalTimeout:                   globalTimeout, skipWaitForDeleteTimeoutSeconds: d.SkipWaitForDeleteTimeoutSeconds, out:                             d.Out,

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: // Get entry, get mount options at 6th word, replace brackets with commas command += fmt.Sprintf(" && ( mount | grep 'on /mnt/test' | awk '{print $6}' | sed 's/^(/,/; s/)$/,/' | grep -q ,%s, )", option) } <ins> command += " || (mount | grep 'on /mnt/test'; false)" </ins> runInPodWithVolume(client, claim.Namespace, claim.Name, t.NodeName, command) By("checking the created volume is readable and retains data")

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: CRI-O job in our CI using kubernetes master tests Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with mount options Since when has it been failing: since this PR has been merged: Testgrid link: link to our job is here: https://openshift-gce- https://openshift-gce- https://openshift-gce- Reason for failure: we don't know the reason of the failure but it doesn't seem related to CRI-O. It's constantly reproducible also. Anything else we need to know: /cc <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
/sig node /sig storage
/cc
Curiously, it does not seem to have flaked in core kubernetes CI
I'm still looking at it on our side
This test is newly in this commit. It tests that nfs external dynamic provisioner can provision a volume with specific mount options and the volume is properly mounted in the pod. The mount options specified in this test are "vers=4.1", "proto=tcp", and "nosuid". () The test seems to fail to create a pod with the volume and other tests that don't specify these mount options don't fail. So, I guess that this is caused because this environment doesn't support either of the option above. Is it possible to try testing it manually by changing these options?
yes, I have a box where I can change them and re-test, do you have suggestions on what to use to test?
(sorry fat fingers)
If it was an unsupported mount option, then the test should fail every time, not just flake?
yes, it's failing every time for us
Are there kubelet node logs we can look at?
yes, we have artifacts for each job at https://gcsweb-
As a starting point, I would recommend deleting only "proto=tcp", and check if it still fails, first. Then moving back "proto=tcp" and deleting only "nosuid", and check what happens. And continue testing with combinations of options.

Or, if we can find harmless mount options that can be specified for nfs, we will be able to replace the options with certainly harmless ones. Note that tests for AWS and GCE uses "debug" and "nouid32", which don't seem to be valid. for nfs
The container is failing with exit-code 1 with: I'll try to see which one is the bad one
do you think we could make the test easier to debug and figure out which mount option doesn't work?
yes, I'm trying to add something to that command to at least have some logs if that fails cause right now, if you do kubectl get logs on the failing pod you don't get anything
looks like removing makes the container happy and test as well. I'm not sure why it fails on rhel and fedora though, I'll dig into that, but yeah, we could use something different if possible
confirmed that if I remove nosuid the test works fine, I"m modifying the test code to output something useful in case it fails
Thank you for your quick investigation and improvement!
I'm going to open a PR for the test output to have something useful () and understand why it fails with
sorry, I am the one who wrote that hideous bash one-liner. TY for any patches
so, isn't shown: is that required for the test? can we have another option checked instead?
maybe we can use ? who is setting and where?
PTAL at
do you guys have any insight on this?
As for this e2e test, we will be able to use any options which are valid for nfs e2e test, because the purpose of the test is to check that the volume mount options work well for provisioning and the options are properly reflected to the mount options inside the pod. (Actually, I chose for nfs test just because I found it first.) As for issue for nfs dynamic provisioner with this environment, it might be still worth looking into, if this option can be specified by users. However, this could be out of the scope of this issue.
exactly, thanks, I'm going to investigate it not as part of this PR
Since this test failure is non-blocking for release, setting priority approprately. /priority important-longterm


Question: cmdIP6Tables        string = "ip6tables" ) <del> // Option flag for Restore </del> <ins> // RestoreCountersFlag is an option flag for Restore </ins> type RestoreCountersFlag bool <ins> // RestoreCounters a boolean true constant for the option flag RestoreCountersFlag </ins> const RestoreCounters RestoreCountersFlag = true <ins> // NoRestoreCounters a boolean false constant for the option flag RestoreCountersFlag </ins> const NoRestoreCounters RestoreCountersFlag = false <del> // Option flag for Flush </del> <ins> // FlushFlag an option flag for Flush </ins> type FlushFlag bool <ins> // FlushTables a boolean true constant for option flag FlushFlag </ins> const FlushTables FlushFlag = true <ins> // NoFlushTables a boolean false constant for option flag FlushFlag </ins> const NoFlushTables FlushFlag = false <ins> // MinCheckVersion minimum version to be checked </ins> // Versions of iptables less than this do not support the -C / --check flag // (test whether a rule exists). var MinCheckVersion = utilversion.MustParseGeneric("1.4.11") <ins> // RandomFullyMinVersion is the minimum version from which the --random-fully flag is supported, // used for port mapping to be fully randomized </ins> var RandomFullyMinVersion = utilversion.MustParseGeneric("1.6.2") <del> // Minimum iptables versions supporting the -w and -w<seconds> flags </del> <ins> // WaitMinVersion a minimum iptables versions supporting the -w and -w<seconds> flags </ins> var WaitMinVersion = utilversion.MustParseGeneric("1.4.20") <ins> // WaitSecondsMinVersion a minimum iptables versions supporting the wait seconds </ins> var WaitSecondsMinVersion = utilversion.MustParseGeneric("1.4.22") <ins> // WaitRestoreMinVersion a minimum iptables versions supporting the wait restore seconds </ins> var WaitRestoreMinVersion = utilversion.MustParseGeneric("1.6.2") <ins> // WaitString a constant for specifying the wait flag </ins> const WaitString = "-w" <ins> // WaitSecondsValue a constant for specifying the default wait seconds </ins> const WaitSecondsValue = "5" <ins> // LockfilePath16x is the iptables lock file acquired by any process that's making any change in the iptable rule </ins> const LockfilePath16x = "/run/xtables.lock" // runner implements Interface in terms of exec("iptables").

Anyone itching to understand the code and try fixing things in Kubernetes, please help fixing the lint failures Step 1: Remove one or two packages (related packages) from the list in - (note: do not change generated files, the name of defaulting functions, or the name of conversion functions... packages failing linting due to issues in those files will need to be left alone) Step 2: Run and look at the logged errors Step 3: Fix the code to ensure that runs cleanly. Step 4: Run and to make sure we fix any problems with gofmt or bazel. Step 5: File a PR with the change Step 6: Go to step :) Please use the tracking spreadsheet For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
/sig architecture /sig testing
Hey, I would like to resolve this issue. Could you assign it to me ?
there are a lot of packages so there will be many people working on this. Also we cannot assign issue to folks who are not in the github org for kubernetes. please feel free to pick up a few packages like i mentioned and file a PR. Make sure you link back to this issue, so folks can see which packages you filed PR for.
OK. I see. I will firstly pick up the package called to fix.
Hi dims. Should i submit my PR before i remove one or two packages (related packages) from the list in - Or I first remove the package from list, then I fix the issue and submit a PR?
in your PR, you will have both the fixes to packages and some lines removed from /.golint_failures
When I run , I found some errors as below: I found that there is a go file in But there is still a error.
Say your GOPATH looks like this Your kubernetes directory should be () cd to and then run from that directory.
Is it necessary to obey all rules in golint? Rules in golint are set in stone. Maybe using gometalinter to ignore some specific errors is better?
I revise some erros about golint in some packages. Please review it.
which errors would you like to ignore?

if you can open a new issue with the list of errors to ignore (and proposing gometalinter), we can get some opinions there from others in the community.
Let me take the kubectl cmd part...
Can someone please review and let me know if I am doing it right?
I ll work on all the packages related to
we get 800-ish pkgs left to fix. my concern is that if we fix them package by package, then such pulls will be flooding the repo. can't we possibly make it in larger pulls? the larger the better?
+1 to slightly larger pulls with related packages. if allow too large then the reviewers will not be able to do a thorough review. we should spread the load among multiple reviewers. (if we allow a really large pull, we will end up with top level reviewers who will face the burden and the PR(s) will just stagnate)
oh good point. we could put one pkg's change into one commit, and merge all changes under one area into one pull so that it wont spread the load. wdyt? for each pull, we can have hundred-ish lines of change. but it will still be 30~50 pulls i guess? good news is that we can take our time to deal with it, at least before 1.12
sounds great! please feel free to help make it happen :) (by suggesting groups of packages that kinda go together)
I'm looking at fixing pkg/proxy, and I think I see what was talking about ignoring some rules. proxy.ProxyProvider is being suggested because golint is saying that it stutters. In that example, I think it's okay renaming it to proxy.Provider, but I could see an example where you would want to keep the stutter if the package that it's declared in uses it heavily.
sure thing, feel free to propose a PR for switching to gometalinter independently of other patches. We can work through that there.
I sent the pull request to fix pkg/volume/aws_ebs
I've fixed a few of these, but get the following error when I try to run : Are there any instructions on what I need to setup in order to resolve this?

you can find this file in
I have raised a PR to fix golint errors in pkg/api
I have fixed some pkgs under /pkg/kubelet pkg/kubelet/images pkg/kubelet/kuberuntime pkg/kubelet/leaky pkg/kubelet/pleg
I submitted a pull request for golint errors in pkg/security/ edit: I can work on picking up pkg/controller/bootstrap through pkg/controller/namespace/... (114-129)
fix golint errors in package cmd/kubeadm/app, since the codes under cmd/kubeadm/app/apis/kubeadm/v1alpha2 and cmd/kubeadm/app/apis/kubeadm/v1alpha3 are auto-generated, i keep them untouched.
I am working on some packages under pkg/kubelet, anyone interested in this plz leave messages for me :)
fix golint for pkg/util/dbus pkg/util/labels pkg/util/node pkg/util/strings pkg/util/workqueue/prometheus
PR fixes golint errors in pkg/routes,pkg/securitycontext, pkg/serviceaccount
PR fixes golint in pkg/probe
should we have a checklist as part of the issue summary saying what is in progress and what is done? Since this issue is getting long, this will make new contributors to pick up issues easier.
Great idea!, want to throw a google doc with a check-list so it's easy for participants to organize themselves? (please!) If it is the issue summary then one of us with privileges will be a bottle neck.
Created a with all the package that have on going PRs. Maybe you can add this to the summary. For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
Thanks updated the spread sheet
Raised PR to fix golint issues in: pkg/controller/clusterroleaggregation pkg/controller/cronjob pkg/controller/daemon pkg/controller/deployment pkg/controller/deployment/util pkg/controller/disruption pkg/controller/endpoint
This issue is close in accident by . could you reopen it?
/reopen
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I found a problem trying to fix golint messages for .

Golint complaints on 's function because of the snake case. When I try to fix it, fails because it is being used in a code generated file. If I clean generated files, tests pass, but the generated file is not invoking it anymore. I failed to find the generator definition using this so I could change the hook to just . Also I'm not sure if this is wanted. I also found other packages in also probably affected by this. I read somewhere about replacing golint with gometalinter, which would allow to define directives in code to omit certain rules. I would consider to have a second exclusion file for golint with these cases, to be updated if move to gometalinter is finally done. This would make this issue completion more feasible. Meanwhile, I'll move onto fixing golint messages for other packages, until this discussion gets cleared.
I also found a problem fixing . A test fails because the directory is marked as readonly (see tests @ Shall we also ignore this directory for this issue?
Is this issue still relevant because hack/verify- is giving me no errors.
the failures are still present in there are details in the first message on how to claim + solve issues :)
Sorry I misunderstood the issue
I'm working on this package:
pkg/apis/admission/v1beta1 pkg/apis/authentication/v1beta1 pkg/apis/authorization/v1beta1 pkg/apis/coordination/v1 pkg/apis/coordination/v1beta1 pkg/apis/imagepolicy/v1alpha1 pkg/apis/scheduling/v1alpha1 pkg/apis/scheduling/v1beta1 pkg/apis/settings/v1alpha1 pkg/apis/storage/v1alpha1
pkg/cloudprovider/providers/vsphere/vclib pkg/kubelet/apis/deviceplugin/v1alpha pkg/kubelet/apis/pluginregistration/v1alpha1 pkg/kubelet/apis/pluginregistration/v1beta1
:
I'm looking at the pkg/registry/*, all those which are left. I expect to give an update on what all is fixed in the next 24-48 hours.
hi there is a google sheet that is a bit outdated, maybe you can update that?
hi I seem to only have view access for the sheet. Can't edit.
I fixed the permission.

Please try it again
pkg/apis/authentication pkg/apis/coordination pkg/apis/events pkg/apis/imagepolicy pkg/apis/settings
hi I think this issue was closed prematurely by (using fixes #issue in commit)
I'll put in a PR shortly for the package
Looking at next
hi thanks! there is a spreadsheet here that tracks the progress on fixing these issues, consider adding the packages you're work on there so we can keep everything organized :+1:
Starting on now
I did indeed see the Google Sheet link (thank you and to take it a little further
test/e2e
test/e2e_node/builder
pkg/registry/admissionregistration/mutatingwebhookconfiguration/storage pkg/registry/admissionregistration/rest pkg/registry/admissionregistration/validatingwebhookconfiguration/storage pkg/registry/apps/daemonset/storage
Hi. I opened a PR - Please review.
Working on Will open a PR soon.
Starting on and should have a PR soon
I keep getting this golint verify error on and was wondering if others saw it: But when I that package to the golint failures in my PR, my reviewer said it isn't necessary? What does that mean? What did I miss? Should I use a different kind of verify? cf:
looks like the disconnect is because that problem does not show up in the version of we use in the CI. please see
Still confused: is that not the golint version I ran? it installs from vendor and uses that I think?
i installed from and i still don't see the problem, can you please check what version of golint you have?
I'm confused why I wouldn't be using the vendored version? how would that get out of sync?
/me looking
Note: I am reserving for a KubeCon Contributor Workshop demonstration. It is noted in the spreadsheet.
I think may not be ideal cause there's a lot of auto generated files there. What do you think about reserving instead for the KubeCON Contributor Workshop?
pkg/apis/scheduling pkg/apis/storage/util pkg/apis/storage/v1beta1/util pkg/apis/storage/v1/util
/priority backlog
Hi I am a new contributor here, I'd like to contribute and make changes to the following file/s. I have marked it as WIP in the Google Sheet. Can someone let me know if I have to do anything else before I start working on this PR?

Umm so there's an issue where Golint tells that underscore shouldn't be allowed. Here's a log of the go-lint warnings:
Thanks The following should not be modified, as they are designed like this for code generation.
I'd like to pick a package and do my first PR, is there a list of packages to be done? Or should I correlate with this issue thread? Thanks in advance!
please see step 1 () please scan other PRs linked to this issue as well just in case.
Will do. Thank you. On Thu, Oct 3, 2019, 7:25 AM Davanum Srinivas <:
Hi there, I'm looking for a good first issue. I'll start taking a look at if that's ok? There doesn't seem to be anyone currently tracked as looking at it in the attached spreadsheet.
I've found a few others I'm actively looking at, you won't get any argument from me. On Sat, Oct 12, 2019, 10:38 AM Pamela Kelly <:
I am working on the iptables pkg. Updated in the google docs
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have problem running in step 2 $ hack/verify- what should I do?
PTAL for my pr to this issue, thx
you can move kubernetes directory to and re-run this script.
I'd like to take Can we fix linter issues for generated files? Or we should skip packages with zz_generated?
There're 2 lint errors that affects 292/507 packages, meaning it's not possible to cleanup the entire file without solving it: default pattern of Kubectl is the usage of [cmd]Options for initial options structs and it raises: /apis/{custommetrics,externalmetrics} for example raises the underscore error, but It's probably not worth to break backward compatibility renaming these folder for lint error fix: Any advice on these?

/reopen hm, cannot reopen this issue, it was closed by mistake
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am sure this is all not fixed. Am I missing something? ./hack/verify- installing golint from vendor Congratulations! All Go source files have been linted.
you need to remove package from first.
/remove-sig testing SIG testing doesn't own code failing lints, we own the linter. :upsidedownface:
Hello, is this still a relevent issue ? I would like to work upon
I have fixed
Hi Folks, Some of the golint fixes, require us to change the exposed function's name, in simple words, "change API". I was not sure, if that is welcomed and what is the process.
Heres what Im getting after going over the following steps as mentioned by Say your GOPATH looks like this GOPATH=/Users/dims/go Your kubernetes directory should be ($) cd to and then run hack/verify- from that directory. I get the following output - installing golint Congratulations! All Go source files have been linted. Im not really sure what I am doing wrong. Any help is appreciated!
I think I understand. I misunderstood the steps around .golint_failures. The first step is to remove the pkg that we are targeting from this file and then run the My mistake was to assume that the packages in the file need to be removed after we are done cleaning the golint errors.
Can someone please review and let me know if I've done this right?
After a couple years of seeing changes spawned from this issue, I don't think this is a good use of contributors' or reviewers' time.

golint is very opinionated about: exported type and function names, leading to PRs attempting to change exported things people use downstream missing documentation on interface implementation functions, which leads to not-very-useful copy/paste of interface godoc the govet and staticcheck verify jobs are much more useful since they catch correctness issues, but I would recommend closing this issue and dropping the golint checks from CI golint doc itself discourages from using its output in an automated enforced way: from
i guess one can say gofmt is opinionated too, but something good about Golang is that it comes with an opinion and a set of standard tools. for API documentation at least we found inconsistencies on how most exported API items are documented: golint can help with that, but if there are no "pragmas" to avoid breaking changes and given we have codegen generating non-compliant names enabling golint on API packages is sort of blocked in Kubernetes. for non-API, non-library, non-public code, i think being inconsistent is mostly OK. no preference on my side. shrug
true, but code formatting has the benefits of mostly eliminating meaningless diffs in PRs due to code movement/whitespace/etc if there are specific packages (like API packages) we lack documentation on, adding checks specifically for that could make sense exactly, I don't think golint is the right tool for making sure we have API docs
golint itself doesn't have these and the go project is against this sort of comment AIUI, however golangci-lint does have them. I think there are plenty of other static analysis lints we should use instead, beyond just staticcheck and vet there is e.g.
btw, if we remove golint can we see e.g. all-caps constants or functions with underscores dodging review? perhaps we should add golangci-lint soon after:
Completely agree there are better places to spend review bandwidth. I was curious to see what the effort looked like. There have been 890 commits that touched over the past 3+ years !
/remove-help /remove-good-first-issue
Overall agree on the bandwidth. Also note that it was a good first issue with very low bar to get them in the repo. May be we need to find a similar low bar, but high value good first issue.

UT's come to mind, are there other things we curate for new contributors ?
Do we have an alternative way of enforcing that new exported code (not necessarily API fields) has at least some minimum documentation? As a reviewer, I ask contributors to add meaningful comments. But reality is that I sometimes stumble into old code that doesn't have documentation because we didn't have golint at some point. I agree that fix-golint PRs just add burden to reviewers. But, before removing the lint check, can we wait and see if it's enough to discourage people from sending more PRs like this? Additionally, we can encourage reviewers to just close PRs that only fix golint issues.
+1 for adding burden to reviewers. However, as a new contributor, I think this a very good opportunity for new contributors to study the code base and submit their first PR. In my opinion, I think this issue would be able to attract more new contributors, get familiar with the code base, and get familiar with the git workflow.
thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: At some point we can look at adding golangci-lint, there's a WIP in progress in /close
Closing this issue. <detailsIn response to : thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // When no endpoint slices would usually exist, we need to add a placeholder. if len(existingSlices) == len(slicesToDelete) && len(slicesToCreate) < 1 { <ins> // Check for existing placeholder slice outside of the core control flow </ins> placeholderSlice := newEndpointSlice(service, &endpointMeta{Ports: []discovery.EndpointPort{}, AddressType: addressType}) <del> slicesToCreate = append(slicesToCreate, placeholderSlice) spMetrics.Set(endpointutil.NewPortMapKey(placeholderSlice.Ports), metrics.EfficiencyInfo{ Endpoints: 0, Slices:    1, }) </del> <ins> if len(slicesToDelete) == 1 && placeholderSliceCompare.DeepEqual(slicesToDelete[0], placeholderSlice) { // We are about to unnecessarily delete/recreate the placeholder, remove it now. slicesToDelete = slicesToDelete[:0] } else { slicesToCreate = append(slicesToCreate, placeholderSlice) spMetrics.Set(endpointutil.NewPortMapKey(placeholderSlice.Ports), metrics.EfficiencyInfo{ Endpoints: 0, Slices:    1, }) } </ins> } metrics.EndpointsAddedPerSync.WithLabelValues().Observe(float64(totalAdded))

My cluster contains about 4w endpointslices. There are 3 related metrics (endpointslicecontrollerendpointsaddedpersynccount, endpointslicecontrollerendpointsremovedpersynccount, endpointslicecontrollerchanges) to indicates endpoint add remove and update. I found that the apiserver periodically receives lots of endpointslice update requests. In terms of the number of requests, the endpointslice has been fully updated. But from the endpointslice resource itself, it seems that it has not been updated, because "" annotation didn't change It seems that there is no need to update the endpointslice with no really changes. On the other hand, a large number of ep slice update requests will stress the apiserver. In order to make the phenomenon more obvious, the min-resync-period parameter of controller-manager can be adjusted to 1 minute. Create one or more empty endpointslices, and then you can find in the apiserver logs that requests to update these endpointslice every one to two minutes. For clusters with lots of empty endpointslices, this bug may affect the stability of it. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
there were different improvements in this area, but since 1.20 is going EOL in 8 days it will be good to know if this happens in 1.21 /cc
I'm glad to recieve your comment. I'm not sure because I only have cluster with version 1.20.10. Expect someone who could provide more detail about it. It is actually happening in my cluster, but I don't know how it happens.
I've seen this in 1.20 clusters as well, but have not seen it in 1.21.

My theory was that this is mostly or entirely fixed by Prior to that, any change to EndpointSlice labels would not trigger an EndpointSlice controller sync loop, but the resync period would result in those EndpointSlices being updated at that point. more recent fix () seems like it could also be relevant, but I don't think it matches what's described here. In that case, I believe we're unnecessarily updating EndpointSlices if something irrelevant in the Pod changes (resource version), but I don't believe there's anything special there that would mean we weren't updating EndpointSlices until the resync period. So with all of that said, I believe this is fixed in 1.21+, but let me know if anyone's able to recreate this in those versions and/or if is actually more relevant than it appears on the surface.
Thanks for your reply I will check it.
I confirmed that the periodic updates in my cluster are all empty endpointslices. There are l lot of IPv6 endpointslice in my cluster. But it shouldn't be updated when endpointslice-controller couldn't list pods by service label selector.
/assign
I found that when endpointslice's endpoint array is empty, endpointslice-controller updates it every periodic resync, although it didn't change. I consider it a bug and would like to fork and fix it. The code below could fix the issue of updating empty endpointslices, but I'm not sure if it will introduce new issues. for i := 0; i < len(slicesToDelete); { if len(slicesToCreate) == 0 { break } sliceToDelete := slicesToDelete[i] slice := slicesToCreate[len(slicesToCreate)-1] // Only update EndpointSlices that are owned by this Service and have // the same AddressType. We need to avoid updating EndpointSlices that // are being garbage collected for an old Service with the same name. // The AddressType field is immutable. Since Services also consider // IPFamily immutable, the only case where this should matter will be // the migration from IP to IPv4 and IPv6 AddressTypes, where there's a // chance EndpointSlices with an IP AddressType would otherwise be // updated to IPv4 or IPv6 without this check.

if sliceToDelete.AddressType == slice.AddressType && ownedBy(sliceToDelete, service) { slicesToCreate = slicesToCreate[:len(slicesToCreate)-1] // We must consider the situation that an empty unchanged endpointslice no need to update. If the one to be deleted and the one to be created are the same endpointslice which is empty, skip the update step. // eg: If there are many headless svc with single stack ipv4 for example, a lot of ipv6 empty // endpointslice update could be avoid. if slice.Name != sliceToDelete.Name len(slice.Endpoints) != 0 { slice.Name = sliceToDelete.Name slicesToUpdate = append(slicesToUpdate, slice) } slicesToDelete = append(slicesToDelete[:i], slicesToDelete[i+1:]...) } else { i++ } /cc
Hey I'm not sure I completely understand this fix, but thanks for volunteering to take this on! As I recall, this specific section of the reconciler code is part of the "recycling" logic, where an EndpointSlice marked for creation reuses the name of when marked for the deletion, and the operation becomes an update. I'd appreciate a PR if you can also include a test that recreates the buggy behavior + shows that this fixes it.
Thanks for your reply. I will try more test and show this fix does work.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I understand the bug now - it looks the root issue is here: Essentially we'd consider the previous placeholder EndpointSlice no longer desired, mark it for deletion, and then that would mean no EndpointSlices would be left, triggering a new placeholder slice to be marked for creation.

The recycling logic you linked above essentially merges creates and deletes into updates, resulting in the behavior you described of empty EndpointSlices always being updated during the resync period. I think this should be relatively straightforward to fix with some changes to the code I referenced above.
/assign
GitHub didn't allow me to assign the following users: spencerhance. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: securityGroupID, err = c.ensureSecurityGroup(sgName, sgDescription, getLoadBalancerAdditionalTags(annotations)) if err != nil { klog.Errorf("Error creating load balancer security group: %q", err) <del> return nil, err </del> <ins> return nil, setupSg, err </ins> } <ins> sgList = append(sgList, securityGroupID) setupSg = true </ins> } <del> sgList = append(sgList, securityGroupID) </del> } extraSGList := getSGListFromAnnotation(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups]) sgList = append(sgList, extraSGList...) <del> return sgList, nil </del> <ins> return sgList, setupSg, nil </ins> } // buildListener creates a new listener from the given port, adding an SSL certificate

What happened: Create SG outside of K8s - a SG rule has a range of ports which are opened via K8s Add this into Ensure load balancer switches from the range back to individual ports, the CIDR will be 0.0.0.0/0 if loadBalancerSourceRanges was removed too What you expected to happen: K8S doesn't modify the SG How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.7 Cloud provider or hardware configuration: EKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig aws
Note - I'm not sure if this is related to the aws cloud config, which I can't change as I'm using EKS Add aws cloud config: [global] disableSecurityGroupIngress = true
/sig cloud-provider
We're also having this issue which makes kinda unusable. shouldn't ensure permissions if an SG is a custom one here . could you please take a look?
This is also an issue for us, using these service annotations should have been straightforward. A custom security group id used in the should ensure it's attached to the ELB/LB, and remain untouched. Instead, this customer security group gets purged, leaving the port mapping untouched, but all sources become 0.0.0.0/0.
unfortunately I can't, I don't have access at AWS resources to work on that. To be honest, it was pointed out a number of times that the load balancer code will need a refactor and better tests, I would suggest pinging AWS directly or bringing it up during the meetings.
Suppose you have an existing EKS cluster, gateways, the whole nine yards... Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs. You expect, by using this SG id (sg-) in conjunction with this service annotation that: All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one).

All entries in your custom security group are intact, and would be retained by the control plane. Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules. Temporary solution: Run the override script (in our case, assign the serviceAnnotations to the gateway). Still use a custom security group, but leave it empty. When all and done, add your custom entries to the SG. As of 08/22, this cheap workaround, works. Still needs investigating.
Any subsequent updates to the cluster, reverts all configurations back to 0.0.0.0/0. EKS was updated from eks.2 to eks.3. Which somehow triggered the changes with istio.
Hi, It looks I have the same issue. Company policy dictates that all SG's have a specific name. Before: we just had service deployment which result in an auto-generated ELB with an auto-generated SG with the 'wrong' name, but everything was working fine. I'm now in the process of getting the correct name. So I created a custom SG "A" which has ingress rules SG "A1" and "A2". I add this with Now the elb is still auto-generated (as expected), but it only has 1 of the SG's from "A", being "A2" and a autogenerated one. I the a file with contents in the location. But I cannot get it to work. We use as , which probably creates this extra rule. I cannot find ( or am probably not able to read correctly) the correct config. Should the aws-load-balancer-internal be used or not? Do I need another directive? Maybe someone can help me out with some pointers? Thanks in advance.
It's possible to work around this by setting to something bogus (i.e. ) and putting your desired SG in the annotation instead (k8s won't modify this one).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleDownExcludeUnscheduledPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  5, expectedReplicas: 1, podReadiness:     []v1.ConditionStatus{v1.ConditionTrue, v1.ConditionFalse, v1.ConditionFalse, v1.ConditionFalse, v1.ConditionFalse}, podPhase:         []v1.PodPhase{v1.PodRunning, v1.PodPending, v1.PodPending, v1.PodPending, v1.PodPending}, resource: &resourceInfo{ name:     v1.ResourceCPU, requests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, levels:   []int64{100}, targetUtilization:   50, expectedUtilization: 10, expectedValue:       numContainersPerPod * 100, }, } tc.runTest(t) } </ins> func TestReplicaCalcScaleDownIgnoreHotCpuPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  5,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the HPA scales up beyond the capacity of the cluster, many unscheduled pods are created. When load is removed, the usage ratio drops and the HPA tries to scale down. The unscheduled pods do not produce metrics, so the HPA fills in the 100% usage and re-evaluates the usage ratio. Because of the many fake 100% metrics, the new usage ratio is above the target value and the direction of scale is reversed. In this case, the HPA decides not to make any change to the current scale. What you expected to happen: I would expect the HPA to not get stuck when load is removed. How to reproduce it (as minimally and precisely as possible): and put it under load in a cluster without cluster autoscaling. the Deployment to increase the CPU request beyond the capability of the cluster (e.g. 500m). This should produce many Pending pods. load. The Pending pods will keep the HPA scaled indefinitely. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.9-gke.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T21:30:53Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig autoscaling
This might Previous I suspected that this was the same issues as (HPA positive feedback loop).

However it turns out to be different, that Pending pods cause HPA to get stuck.
Original reason for using 100% utilization for missing metrics when scaling down:


Question: } c.invalidateCachedPredicatesOnNodeUpdate(newNode, oldNode) <del> c.podQueue.MoveAllToActiveQueue() </del> <ins> // Only activate unschedulable pods if the node became more schedulable. if nodeSchedulingPropertiesChanged(newNode, oldNode) { c.podQueue.MoveAllToActiveQueue() } </ins> } func (c *configFactory) invalidateCachedPredicatesOnNodeUpdate(newNode *v1.Node, oldNode *v1.Node) {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: t.Errorf("expected success or bad request err, got %v", err) } }) <ins> t.Run("JSONPatchType should handle a valid patch just under the max limit", func(t *testing.T) { patchBody := []byte(`[{"op":"add","path":"/foo","value":0` + strings.Repeat(" ", 3*1024*1024-100) + `}]`) err = rest.Patch(types.JSONPatchType).AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")). Body(patchBody).Do().Error() if err != nil { t.Errorf("unexpected error: %v", err) } }) </ins> t.Run("MergePatchType should handle a patch just under the max limit", func(t *testing.T) { patchBody := []byte(`{"value":` + strings.Repeat("[", 3*1024*1024/2-100) + strings.Repeat("]", 3*1024*1024/2-100) + `}`) err = rest.Patch(types.MergePatchType).AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It returns an error when I create a pod of yaml file over 1M. I checked the codes and found that the coded wrong. It should be “json.Unmarshal(p.patchBytes, &v)”, not “json.Unmarshal(p.patchBytes, v)” in the 344 and 368 line. What you expected to happen: It can be parsed correctly. How to reproduce it (as minimally and precisely as possible): Look at the code and you'll see or you can test it. Anything else we need to know?: no. Environment: Kubernetes version (use ): Community latest code Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind bug
/sig api-machinery
/assign


Question: var _ volume.Mounter = &azureDiskMounter{} func (m *azureDiskMounter) GetAttributes() volume.Attributes { <del> volumeSource, _ := getVolumeSource(m.spec) </del> <ins> readOnly := false volumeSource, err := getVolumeSource(m.spec) if err != nil && volumeSource.ReadOnly != nil { readOnly = *volumeSource.ReadOnly } </ins> return volume.Attributes{ <del> ReadOnly:        *volumeSource.ReadOnly, Managed:         !*volumeSource.ReadOnly, </del> <ins> ReadOnly:        readOnly, Managed:         !readOnly, </ins> SupportsSELinux: true, } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I'm running a Redis pod with a PV provisioned as an Azure Managed Disk. It's got an exec readiness probe that calls in the container. At some point it receives a SIGTERM. The kubelet then tries to sync the pod and restart the container, but it panics in (looking over the code, that's the only place where the error from is ignored) As a result, the pod hangs in a state, since the readiness probe cannot be ran in a non existing container. What you expected to happen: The container would successfully restart. How to reproduce it (as minimally and precisely as possible): It's the 2nd time it happens but I haven't yet figured out the conditions to reproduce. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: Azure w/ Managed Disks OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi could you also share the storageclass, pvc & pod yaml files? I would like to repro, thx
I may insert a nil check statment in this func: But I need to reproduce this bug anyway, so that I could find the root cause of this issue, thx.
Yes, an error check is what I'm hoping for :) I've used the chart. The PV/PVC were created manually, they're set to an empty storage class. I should mention that I've also noticed other pods being terminated/recreated at the time this one failed, however I didn't trace the cause of this yet.
thanks for the reporting, I may have found the root cause, just need a few more debuging time, thx.
GitHub didn't allow me to assign the following users: xiazhang. Note that only can be assigned. <detailsIn response to : /assign xiazhang Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign andyzhangx
- Thanks for the fix!


Question: <ins> ### debian-hyperkube-base Serves as the base image for `gcr.io/google-containers/hyperkube-${ARCH}` images. This image is compiled for multiple architectures. #### How to release If you're editing the Dockerfile or some other thing, please bump the `TAG` in the Makefile. ```console # Build for linux/amd64 (default) $ make push ARCH=amd64 # ---> gcr.io/google-containers/debian-hyperkube-base-amd64:TAG $ make push ARCH=arm # ---> gcr.io/google-containers/debian-hyperkube-base-arm:TAG $ make push ARCH=arm64 # ---> gcr.io/google-containers/debian-hyperkube-base-arm64:TAG $ make push ARCH=ppc64le # ---> gcr.io/google-containers/debian-hyperkube-base-ppc64le:TAG $ make push ARCH=s390x # ---> gcr.io/google-containers/debian-hyperkube-base-s390x:TAG ``` If you don't want to push the images, run `make build` instead [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/build/debian-hyperkube-base/README.md?pixel)]() </ins>

I'm pretty sure that building the hyperkube images takes longer than building kubernetes itself now. At least a large part of this seems to be that we download and install ~172 extra debian packages several times (once for each arch). <details</detailsNot sure how to improve this - maybe better caching? Do we actually want to install these extra packages? Should we use a better base image than ? cc
cc (not really sure who "owns" hyperkube, if anyone)
The hyperkube binary is one thing. I think the image ends up being big as we drag in so much stuff. I wasn't involved in creating the docker image so I can't comment. Ideally this thing would be slim and would have minimal external dependencies. Fixing this starts with mapping why we have each of those deps imported.
I actually have written a such patch in my local env, but I never uploaded. It removed the docker and docker-multinode-specific stuff and all the unused deps. Also, we will come one step closer once is in. Should I submit the patch so you can take a look at it?
sure!
This needs to be triaged as a release-blocker or not for 1.5
punting to v1.6.
The main problem here is that we're installing huge packages for supporting some volume types: git nfs-common glusterfs-client cifs-utils With containerized volume mounters we'll get rid of that.
When it comes to local rebuilding, should help here. As I've already noted in the PR, maybe we should make CACHEBUST=0 the default for all hack/dev- scripts. This would at least ease development and testing.
can this be closed? or moved to 1.7?
should this be in v1.7? is there an appropriate sig to own this?
cries nope, not v1.7. I'm still not sure who owns hyperkube.
Everyone and no one I suppose. Yeah, agree it's sad.
/sig release guess we're saying sig-release owns hyperkube under the general guise of "cutting bits" until/unless there is a better home for it
Yeah, hyperkube definitely belongs to sig-release as it hasn't been owned by a sig so far...


Question: return filepath.Join(plugin.host.GetPluginDir(plugin.GetPluginName()), driver, volSha, globalMountInGlobalPath), nil } <del> func getDriverAndVolNameFromDeviceMountPath(k8s kubernetes.Interface, deviceMountPath string) (string, string, error) { // deviceMountPath structure: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/{pvname}/globalmount dir := filepath.Dir(deviceMountPath) if file := filepath.Base(deviceMountPath); file != globalMountInGlobalPath { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath failed, path did not end in %s", globalMountInGlobalPath)) } // dir is now /var/lib/kubelet/plugins/kubernetes.io/csi/pv/{pvname} pvName := filepath.Base(dir) // Get PV and check for errors pv, err := k8s.CoreV1().PersistentVolumes().Get(context.TODO(), pvName, metav1.GetOptions{}) if err != nil { return "", "", err } if pv == nil || pv.Spec.CSI == nil { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath could not find CSI Persistent Volume Source for pv: %s", pvName)) } // Get VolumeHandle and PluginName from pv csiSource := pv.Spec.CSI if csiSource.Driver == "" { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath failed, driver name empty")) } if csiSource.VolumeHandle == "" { return "", "", errors.New(log("getDriverAndVolNameFromDeviceMountPath failed, VolumeHandle empty")) } return csiSource.Driver, csiSource.VolumeHandle, nil } </del> func verifyAttachmentStatus(attachment *storage.VolumeAttachment, volumeHandle string) (bool, error) { // when we received a deleted event during attachment, fail fast if attachment == nil {

Hello, I've recently been testing the reliability of storage for k8s v1.25.0. A strange phenomenon occurs occasionally during the test: the volume cannot be detached after the PVC and pod are deleted(the reclaimPolicy of pv is delete. So I first went to check the corresponding and objects and found that they both existed. Then I checked the kube-controllerManager and kubelet logs and found that kubelet logs kept printing: But I found that at the beginning, the volume was unmountDevice succeed, as shown in the following kubelet log: it is so wired ! Then I some log variables to the unmountDetachDevices method: The results show that And I've drawn up a time-series chart of the contents of the log. ! Let's go back to the unmountDetachDevices method. According to the above-mentioned log content, the first operation successfully changes of the corresponding attachedVolume to , but the function in the subsequent operation still uses stale DeviceMountState. that the unmountDevice operation is not performed after the first unmountDevice operation succeeds. multiple unmountDevice operations are allowed, ensure that the correct pvname can be obtained in the method. the volume is detached after the pod and PVC objects are deleted This problem can be reproduced by repeatedly deleting and recreating pods that require PVC in k8s 1.25.0 cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage
we can't safely assume that the json is always there.
ping
Thanks for the detailed analysis . I'm still sorting through this, do you think that is involved with this?
Yes. look at this log content: The global mount path is changed in the As a result, the method obtains incorrect pvName(e6104b10fea107e004e3307b0444fd7b775fedd804ab47aa997118b7927c32e0). The unmountDevice operation may be performed multiple times in [1.19 - 1.25] kubernetes versions. This problem does not occur before because can still obtain the correct pvName from the global mount path. However, this problem occurs after is merged. https://user- <br</brAfter the first unmountDevice operation is performed successfully, the file is deleted.

So the subsequent unmountDevice operation will perform method because file doesn't exist.
Awesome, thanks. Let me find someone to work on this.
I ll take a look at the issue, since its related to the change I made Note: this change needs the node to drained before upgrading to a version with this change . Please check the release notes
Thanks a lot !
yep, I always reproduce this problem in a newly created k8s 1.25.0 cluster.
Hihas there been any progress on this issue?
/priority important-soon
/triage accepted
Hihas there been any progress?
I also hit this error after pvc and pod deleted on k8s 1.25: and then these two volumes cannot be unmounted on the node forever: I tried to delete pv, it's stuck in terminating state forever
restart the kubelet on the node in problem could mitigate this issue, but we really need to fix this issue ASAP, I could repro this issue easily when running 1000 disk attach/detach test
I think this is again a TOCTOU bug in unmountDetachDevices.
The Unmount operation is , thus the state may not be set as before next reconcile, so it is possible that unmountDevice operation performed multiple times. So the root cause of this issue is that got wrong PV name, since changed global mount path.
I have found a similar issue in our environment, which is based on Kubernetes 1.25.3. If I install a Pod with a PVC, and I force move the Pod with NodeAffinity around the cluster, around 10-20 retries the Pod does not come up: the VolumeAttachment is bound to the old node and is not moved by the system anymore. If this is the same issue as reported here I do not know, but the logs are similar: From the Worker's kubelet where the volumeattachment is bound to: Also I see lots of these: But, if this is the same, I would like to know that the referenced PR was merged into exactly which Kubernetes version, as I have found in the RELEASE NOTES, that it was in 1.24.0-alpha: But we could not reproduce the issue on Kubernetes 1.24.2. How come?
As per the logs, it's the same issue. Kubelet operates in a reconcile loop, and is an async operation.

This bug only happened when is operated for multiple times, so it cannot be 100% reproduced. However, if the possibility of reproduction in 1.25 is much higher than the cluster lower than 1.25, we should figure out whether there are some other changes that makes this bug happened more frequently.
I have 1K disk attach/detach to 40 nodes test, if it runs multiple times, it could 100% repro, we could use this patch for testing, if 100 test runs pass, then it could prove the fix really works.
Sure. I mean not every single Mount/Unmount operation will lead to this error, but if we do it enough times, then it could be 100% to reproduce.
May I ask what could be a workaround in production systems or even CI systems on this issue? Our CI is struggling with this and other random issues, to the point we would like to stabilize it by kicking the system in a way to fix things... I was trying a kubelet restart on the Worker where the Mount is stuck... but many cases the kubelet restart itself hangs or does not produce a correct restart. So what could be a workaround if we find out that there is a hung volumeattachment?
restart kubelet, and if restart kubelet hangs, reboot the node.
Volumeattachment failed to detach from old node When statefulset application with pvc scheduled on worker1 during deployment its volumeattachment bounds to worker1 when same application upgraded scheduled on another worker node say worker2 and struck in ContainerCreating state and volumeattachment failed to detach from old node .

when pod described: Events: Warning FailedMount 23m (x820 over 6d2h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[ datadir]: timed out waiting for the condition Warning FailedMount 5m22s (x3069 over 6d3h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition Environment: NAME="SLES" VERSION="15-SP4" VERSIONID="15.4" PRETTYNAME="SUSE Linux Enterprise Server 15 SP4" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp4" DOCUMENTATIONURL=""' Kubernetes version: v1.25.3 kernel: 5.14.21-.24.41-default Storage: Rook: v1.9.12 ceph: 16.2.10 logs of kubelet: Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ]

"Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85-
FYI. this issue is fixed in 1.24.14, 1.25.10, 1.26.5, and has been fixed in v1.27.0
Hello , can you have a look at this reconciler run's operationExecutor fails to umount volume due to missing vol- It appears to be Kube v1.22 not .25 to boot is this the same issue?
I think they are the same issue and I have left some comments in
Issue reoccurs in kubelet version


Question: }, "FilterPlugin": { {Name: "NodeUnschedulable"}, <del> {Name: "NodeResourcesFit"}, </del> {Name: "NodeName"}, <del> {Name: "NodePorts"}, </del> <ins> {Name: "TaintToleration"}, </ins> {Name: "NodeAffinity"}, <ins> {Name: "NodePorts"}, {Name: "NodeResourcesFit"}, </ins> {Name: "VolumeRestrictions"}, <del> {Name: "TaintToleration"}, </del> {Name: "EBSLimits"}, {Name: "GCEPDLimits"}, {Name: "NodeVolumeLimits"},

Inspired from Our default Filter plugins have the following order: IIUC, this order was chosen to have the fastest Filters or the most probability of reporting Unschedulable first. Arguably, some are pretty much equivalent and could be reordered. The major problem with this order is that plugin is almost at the beginning. This plugin only reports . A Node that fails this plugin will be evaluated during preemption. But perhaps this Node doesn't satisfy other UnschedulabeAndUnresolvable requirements (taints, node selectors, nodeaffinity). Theoretically, having those Filters run first could improve preemption latency. The only open question is whether or not the ordering we have today is an API guarantee. I believe it's not (at least in component config), because the only way to change the ordering of Plugins is disabling all of them first. Decided order: /sig scheduling
for comment
and :)
Even we changed the order of plugins, the framework.RunFilterPlugins still exits early when returned by plugin. So should we change the current behavior to run the filter plugins until there is one plugin returned ?
That is correct, and that's what it should be.
No, that's what the original conversation suggested and it's likely not ideal in the average case.
+1 to reordering. Taints, affinity, node name can all be executed before checking for resource fit. No, it is not.
+1. Lightweight plugins that may return status can be moved ahead of other plugins that only return status.
FYI: from who also had this concern: So how about , is it also a good candidate to be moved ahead of ? NodeAffinity and NodeSelector both incorporated in , I'm pretty sure NodeSelector logic is simply, but not 100% sure NodeAffinity logic falls into the category of "lightweight".
Let me propose an order, for extra clarity:
internally composes and invokes , which IMO is a bit more expensive than . should be cheaper than . So my 2 cents is:
Both proposals are fine by me.
/triage accepted /help /good-first-issue
/assign
I want to help here :)
/remove-help


Question: } func masterUpgradeGCE(rawV string) error { <ins> env := os.Environ() // TODO: Remove these variables when they're no longer needed for downgrades. if TestContext.EtcdUpgradeVersion != "" && TestContext.EtcdUpgradeStorage != "" { env = append(env, "TEST_ETCD_VERSION="+TestContext.EtcdUpgradeVersion, "STORAGE_BACKEND="+TestContext.EtcdUpgradeStorage, "TEST_ETCD_IMAGE=3.0.17") } </ins> v := "v" + rawV <del> _, _, err := RunCmd(gceUpgradeScript(), "-M", v) </del> <ins> _, _, err := RunCmdEnv(env, gceUpgradeScript(), "-M", v) </ins> return err }

Currently the 1.6 -1.5 downgrade jobs are failing: Looks like this could be for different reasons.
I'll fix quota for the GKE test. It was failing for unrelated reasons yesterday and leaked IPs.
From etcd log: From what I understand, we cannot rollback etcdv3 + proto data store encoding. We can rollback etcdv3 + json data store encoding if we use the updated etcd image that has the rollback tool in it. Details look to be in I'll start playing around with this to see if I can get it working. cc
just merged - will review results in morning before closing
I'm afraid this only uncovers another issue. Last I spoke with he said he was working on something for that.
any luck?
Also is probably needed. Thanks for helping investigate that.
looks like still failing but with different errors this time. I'm seeing: https://k8s- and https://k8s-
thoughts? This is a release blocker
your comment was before merged. Most recent failure is significantly different but I'm finding the output hard to read. I'm talking to to decipher: https://k8s-
I tried to create a 1.6 cluster and then downgrade the master to 1.5, but the downgraded master never responded (waited for hours). The steps I run:
You're hitting the downgrade issue , there is no way to downgrade etcd from v3 storage version to v2. has some env variables that allow you to deploy etcd with v2 for the 1.6 cluster.
We still aren't passing the right variables into the kube-env. In particular, there's no value for , so we end up running the 2.2.1 image which doesn't contain the migration script nor the rollback tool. I can try to reproduce this myself. cc in case we are doing something dumb WRT the environment.
were you able to reproduce?
Looks like is now failing with the same error as , progress! https://k8s- Now it looks like has the static IP allocation issue: https://k8s- could you look into the quota on this one too? It looks like whatever you did fixed the other suite.
I'm not looking at this anymore, ping me if you need me.

I'm willing to count https://k8s- as a pass...
Was decided at burndown that these are test failures and don't need to block 1.6
Doesn't look like we're having quota issues anymore. /unassign
The ingress upgrade test continues to leak static addresses when the disruption fails. The test's func is never called. Please see
This seems to be affecting many PRs at the moment. Is there a clear owner here?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: if cleanupAndExit { return &ProxyServer{ execer:         execer, <del> IptInterface:   iptInterface, </del> IpvsInterface:  ipvsInterface, IpsetInterface: ipsetInterface, }, nil

What happened: Currently kube-proxy cleanup not work, which will left iptables (and ipvs) rules in host and. What you expected to happen: Kube-proxy cleanup can work and not relay on hostName or ClientConnection config. How to reproduce it (as minimally and precisely as possible):
/sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.