Question: done := make(chan struct{}) go func() { f() <del> done <- struct{}{} </del> <ins> close(done) </ins> }() select {

The integration framework runs an apiserver and sometimes other components before each test execution, however, the shutdown is not clean and is leaving a lot of goroutines orphan that keep accumulation between different runs. Some orphan goroutines are connections to etcd, and since the etcd process is shared for all test, we don't know if this can cause contention and flakiness on the job. We are logging the number of goroutines leaked per test binary execution, it is important if we can reduce them and more important we don't regress and this number keeps growing. This issue will record the state as today for reference:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
hmm, not doing better
I'll take the scheduler part for a deep investigate.
For people that want to get access to the stack traces, just apply this patch, so the framework will dump the output of the goroutines after each execution: [x] Apiserver was shutting down the listener before the hooks could finish, hence the controllers couldn't shutdown gracefully since there was no http server [ ] The storage cacher doesn't differentiate between not read and stopping - Missing integration test reproducing the issue, only reproduced with unit tests [ ] admission plugins can't shutdown gracefully - - there is a PR addressing However, most of the contributions to goroutines leaks are grpc connections that are related to the storage cleanup, there is also a TODO in the code and an open issue I've also tried to reduce the number of connections between storage and etcd, right now, each Storage creates an independent conection to etcd, but sharing the etcd client doesn't work
I think I made some progress with it. I was playing with ttlcontroller test (as it's fairly simple one for the playground and I'm familiar with it, but symptoms in many tests are exactly the same) In I actually managed to reduce amount of leaked goroutines from 1023 to 85.

It's not mergable in this state, but it clearly shows that destroy functions for storage are not called in integration tests. I will try to dig deeper into those and clean this up - but I think the above clearly shows where the problem is. - FYI ^^
is out for review and it's making most of the job here There are still leftovers, but those are mostly oneof's here and there - we should address them independently.
And a bit more longerterm cleaner approach is in
With we're down to 14. I admit that this is one of the simplest tests, but apiserver shutdown is pretty clean once that PR merges.
/honk
! <detailsIn response to : /honk Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
a leftover for the storage cleanup in
Re comment above - yeah, I'm aware, we need to clean it up. FTR:
awesome, we are several order of magnitud better
Looking further into it, in different tests I'm seeing a lot of goroutines with the top of the stacktrace like this: should hopefully address those once it merges... ^^
OK - I think I have now the biggest thing contributing factor to the remaining tests, although I don't yet fully understand it and don't know how exactly to fix it. The main problem is basically that TestServer is not finishing its shutdown: So what exactly is happening is: we run TryDown the TestServer there: this underneath is triggering run of PreShutdown hooks: in the meantime ShutdownInitated signal is signaled: as part of running preshutdown hooks, the bootstrap controller is stopping endpoint reconciler: and then as part of doReconcile, the GET requests hangs forever: By adding time.Sleep(5s) (didn't try smaller values TBH) before the ShutdownInitiated is signalled, the TestServer is shutting clearly: Obviously that's not a solution, but I'm wondering why the GET request mentioned above is hanging forever when ShutdownInitiate is signaled and what we can do with this. - thoughts? [The easiest repro is to run: Adding this sleep basically reduces the number of leaked routines from O(1000) to O(20).]

<--- UPDATE ---In fact, setting ShutdownDelayDuration to 5s has the same effect. So there are two main questions: if setting ShutdownDelayDuration is reasonable why the request actually hangs forever instead of finishing with some error... <--- UPDATE ---I think there must have been something different, as I'm no longer able to reproduce the improvement with my current code. Will try to dig deeper. <-- UPDATE ---OK - the last commit in brings it back to fixed.
I thought that the reconciler hangs because of
Not in this place - here the usual GET request hangs - I will keep looking over the next days.
OK - I understand now and the above isn't really true. The request wasn't really hanging forever. What was happening is that it was being rejected by the WaitGroup handler: and retried by the client-go internals... So I guess from that perspective setting non-zero ShutdownDelayDuration actually makes sense.. - thoughts? ^^
mhm, not sure why setting should help. In general, the server shout stop listening and the HTTP std library should instruct the client's stack to tear down the underlying TCP connection. In that case, the client-go should fail on dialing and stop retrying releasing the server. thoughts?
It's that with ShutdownDelay, the "stopCh" is signaling PreShutdown hooks (that still want to do some work, including API calls) and if we immediately start rejecting that, they will not be able to do that work...
Discussed a bit more offline - I'm going to investigate blocking closing the waitGroup filter on preshutdown hooks. Will try to send a PR in upcoming days.
We're getting there - this is where we are now: I have a bit more in my local branch to upstream, but I need to clean them up and may not have time in the next week or so.
where are we standing today?
Getting close:
A bit more coming soon.
This is where we are now: I understand some of them (though fixing is potentially non-trivial) and need to understand the others (in particular serving). And we should prevent regressions by failing if we have more leaking goroutines - will be looking at that over upcoming weeks.

OK - so here is where we are (with the current check set to not leaking more than 4 goroutines): Comparing to the first comment in this issue - that looks like a significant improvement :) Here is the summary of I think the remaining goroutines in the above files: [x] integration/apiserver/tracing: The goroutines are coming from otel exporter (in particular grpc layer). It seems that something is not closed, although required digging deeper into otel exporter. [ ] integration/serving: The goroutines are coming from the failure case: And they basically are coming from the fact that Queue when created is spawning a goroutine: which is only shutdown on Shutdown() call. But Shutdown in all controllers is only called in defer of controllers of the function that is actually not being run in failure cases. The actual queue that is problematic here is coming from here: [x] testing/integration/controlplane/transformation The goroutines are coming from the fact that we're not closing the kms transformer (grpc goroutines) that is being created. We basically create a grpc service and never close that: Fixing would probably require changing transformer interfact to implement Close method or sth like that.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
- can you please help with the apiserver/tracing test mentioned in
looking
Opened to try fixing the goroutine leak.
It seems the fix worked fine - I'm marking the tracing as fixed - thanks I'm also trying to fix couple other smaller things.
I've opened as a final step. With that PR we're in good enough shape that I'm going to close this issue as further improvements would have negligible impact.


Question: // createVolumeSpec creates and returns a mutatable volume.Spec object for the // specified volume. It dereference any PVC to get PV objects, if needed. <ins> // Returns an error if unable to obtain the volume at this time. </ins> func (dswp *desiredStateOfWorldPopulator) createVolumeSpec( podVolume v1.Volume, podNamespace string) (*volume.Spec, string, error) { if pvcSource :=

Deployment/Service manifest: logs from events: In aws, the volume is attached, On the box: In only one is mounted: Where can I debug this further?
Take a look at the on your master for events related to your volume . That will show you when the volume was attached/detached. It appears that your volume attached without issue so there should be nothing unusual here. Then take a look at on the node that the volume is supposed to be attached to. Looks for the UID of your pod (). I suspect one of your mounts is failing. It looks like you are using hyperkube so it might be a mount namespace propagation issue.
I believe this happens when the RC/Pod is being created shortly after the PV or PVC. The latter two appear to need several seconds to be fully available (even if the status is already set to "Bound"). I ran into the same issue with a NFS PV. When I delete and create PV+PVC+SVC+RC all in one go, the associated Pod will be in ContainerCreating state for several seconds (10,<60), then log the warning event and afterwards be created without any problems or further delays. When I first create PV+PVC, wait for several seconds (30) and only then create SVC+RC, the pod will be created immediately and no warning event will be logged. One oddity I noticed is that the PVC will have capacity 0 and empty access modes (as shown by ), even though it was setup to request ROX for 1Gi and the PV provides 1Gi with ROX. This might be a different bug, though.
Thanks for digging into it The capacity 0 issue () was fixed by PR and is in v1.3.5. This is more or less expected behavior. The mount logic has exponential backoff so if it rapidly encounters PVC/PV does not exist, it'll back off the retries resulting in a longer mount time. We can consider optimizing this particular scenario. But I'll lower the priority on this for now and remove the milestone.
Maybe the "Bound" status could be delayed, so the RC would be held back for a while and be scheduled only when the PVC is actually ready?
I don't think the RC actually waits for PVC to be bound before scheduling.

I have also this problem using deployment with HELM, I deploy all my pv,pvc,deploy,svc in one Charts, and my pod who requiere a volume from the PVC get scheduled and stay in ContainerCreating mode. I need to wait minutes or delete pods for they correctly get rescheduled using the volume. Here is the log when the pod get automaticly deleted (after minutes) : It would be awesome if the deployment or RC wait for the PVC to be bound before schedule !
may slightly improve this situation as the maximum backoff time of 2 minutes will not fire anymore.
Note that the same error is thrown when node-size == C5 or M5 for kubectl and with kops . Solution is to set node as t2 machines. Due to it seems.
Reading from the new solution is:


Question: if len(args) != 2 { usageError(cmd, "<pod> and <container> are required for log") } <ins> namespace := getKubeNamespace(cmd) </ins> client := getKubeClient(cmd) <del> pod, err := client.Pods("default").Get(args[0]) </del> <ins> pod, err := client.Pods(namespace).Get(args[0]) </ins> checkErr(err) data, err := client.RESTClient.Get(). Path("proxy/minions"). Path(pod.Status.Host). Path("containerLogs"). <del> Path(getKubeNamespace(cmd)). </del> <ins> Path(namespace). </ins> Path(args[0]). Path(args[1]). Do().

wrote in : This snuck in without namespace support - can you open an issue to support namespace context? Also, how should we handle when we want to use logs for other types of resources (that map to pods, or map to non containers)?
- I can submit a PR to fix this up today if you are backlogged.
Sure, sgtm. Thanks! On Nov 25, 2014 8:35 AM, "Derek Carr" wrote:


Question: <ins> # Copyright 2015 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. FROM scratch ADD mt mt ENTRYPOINT ["/mt"] </ins>

I'd like to commit without e2e support so others can unblock. We need to come back to get e2e on this. I could use help, and it seems like an easy test...
Sorry for my incorrect PR. Now I have amend it.


Question: } } <del> // ResourceEventHandler can handle notifications for events that happen to a // resource. The events are informational only, so you can't return an // error. </del> <ins> // ResourceEventHandler can handle notifications for events that // happen to a resource. The events are informational only, so you // can't return an error.  The handlers MUST NOT modify the objects // received; this concerns not only the top level of structure but all // the data structures reachable from it. </ins> //  * OnAdd is called when an object is added. //  * OnUpdate is called when an object is modified. Note that oldObj is the //      last known state of the object-- it is possible that several changes

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I discovered colleagues writing informer clients that modify objects delivered by informers. I went looking for where it says they should not do that, and I could not find it. What you expected to happen: I expected this critical restriction to be prominently documented. How to reproduce it (as minimally and precisely as possible): Try to find where it says that a client of an informer must not modify an object delivered by the informer. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Documentation aside, I would suggest adding to your CI infra to make it harder for people to make this mistake.
/assign Thank you for fixing docs!


Question: import ( "fmt" "runtime" <ins> "sync" "time" </ins> "github.com/golang/glog" )

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): TPR. GC. OwnerReference. REST mapping. Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT. <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master. v1.5.1 Environment: Cloud provider or hardware configuration: Repro in "hack/local-up-". But I have seen it in many different env. OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The controller manager is unresponsive and overloaded with error message: This error just keeps bloating up without any backoff. This seems that REST mapping didn't recognize the kind. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Generic way: Create a TPR. Create a TPR object "A" Create pods that has OwnerReference pointing to "A". Easy way: Use kubectl create -f kubectl create -f example/example-etcd- Anything else do we need to know:
well, the garbage collector still uses a static RESTMapper(), and in this case, TPRs aren't recognized at all.
Not recognizing is fine and expected. But controller manager keeps on printing out error message without backing off. Can you confirm this is unintended? Or we are missing anything here?
Right. I should have mentioned it: the controller manager is unresponsive and overloaded with error message. Updated in title and top comment.

there are two things that need fixing here: garbage collector should use a dynamic restmapper that picks up types the server knows about, not just compiled-in types a. this means, at minimum, discovering types at start-up b. it also means refreshing at some interval cases where an ownerRef references a type that is not found by the garbage collector's restmapper, it should not panic or hotloop. I'm not really sure what the behavior should be, actually should it treat owners it can't look up as if they were deleted, or ignore ownerRefs for unknown kinds?
If a user runs kubernetes and sees messages like "ignore XXX ownerRef due to unknown kind", he can't really do anything with it. It would be great to recognize the kind, even if there is a delay.
ownerref could have bad data, or it could reference a kind that is no longer part of the cluster. need to define what happens in those cases. when it is referencing a dynamic kind (either a kind contributed by an add-on server or by a thirdpartyresource), yes, it should re-discover on some interval to recognize the kind
I have the same issue when launching an etcd cluster with the coreos etcd-operator. If causes the api server to run out of disk space as it logs infinitely. I have to keep restart the controller container to reclaim disk space. From a security point of view this means a tenant of a cluster can bring the whole cluster down.
Why the target fix is in 1.6 not a patch release for 1.5?
I would not anticipate the garbage collector actually enabling support for TPRs in 1.5, but the bug that causes it to become unresponsive when it sees an ownerRef of an unknown type should be fixed
Changed the milestone to 1.5
I agree. I do not expect the gc to work with TPR either. But it should not "kill" controller manager. And it is the issue we should fix in 1.5. We can make GC work for TPR in 1.6.
For me this is currently a DOS attack
Repro instructions on master in GCE:
so you don't even need to have something with an owner ref pointing at the TPR to trigger this?

I would guess an ownerRef pointing to any arbitrary unknown kind would trigger the same behavior
the etcd operator creates etcd pods with: I suspect is correct.
Ah, the operator triggers it. OK, I see.
etcd operator does not rely on the k8s GC features (it has internal GC). But it does write owner ref. when k8s gc works for tpr, we will remove our temp internal GC. but now the owner ref causes controller manager to spam.
shall we reopen? i do not think this is fixed.
Yeah this just happened to me. Does not seem fixed.
what version are you running? This isn't in a release version yet.
"masks" the problem, not fixes it. We should close this issue when the fix lands.
I just installed with kubeadm two days ago. I believe is 1.5.2. I then tried using etcd-operator and it killed my master with out of disk space writing 300MB/s of log from GC
a fix will be in the 1.5.3 release. It was just merged and hasn't had time to make it into a release. Sure, not arguing that this issue should be closed. I do think is probably good enough for the 1.4, 1.5 branches.
I'll do it after the refactor of GC, , lands. makes gc use the rate limited workqueue, so it will prevents the hotloop. [edit] GC treats unresolveable ownerRefs as though they perpetually exist. That is, an object with an unresolvable ownerRef will not be deleted by GC. It is meant to be a safety net. For example, during a cluster upgrades, if the APIVersion of replication controllers bumps from "v1" to "v2", and assuming all the pods' ownerRefs still have "v1", if GC treats those refs as if the owners were deleted, then all the pods will be accidentally deleted.
Does GC only treat the preferred version as a legitimate ownerRef, or all known versions, or was that just an example?
I guess there is a third option as well, treat unresolveable ownerRefs as though they perpetually exist (which is different from ignoring them).
All known versions.

What I mean in the example is that assuming v1/replicatoncontroller is not supported by the API server anymore.
Actually this is the current case. GC doesn't delete an object with unresolvable ownerRef. I edited my previous comment.
40Gb per day of logs to this mess for us... So we would love a fix too.
is close to get in. It will fix the hot loops. We still need a dynamic restmapper. I'll start a fix after lands.
FWIW, here's the patch I wrote to get back to a more reasonable behavior: (then, rebuild your controller-manager or hyperkube)
Thanks changed gc to use the ratelimited queue, so the hotloop should have been fixed on the master branch. [edit] That said, we still need to make the restmapper dynamic, i'm working on a fix.
My patch was done on the branch, should anyone prefer to avoid upgrading to alpha.0 :)
BTW, is there a backport to 1.5 planned? is flagged so, should a smaller patch be needed, feel free to use this one of course.
Update: I tried to reuse to dynamically translate TPR, but it didn't seem to work because of I'll find a workaround.
what version are you on? said was landed in v1.5.3, so the problems should be masked. Is that true?
I had to overwrite hyperkube in the docker image
Ahhh it has this commit which says: Adjust global log limit to 1ms So probably not a difference a human can detect.
I see. 1 ms is still too spamming.
At least, it make the log unusable without filtering ;) I have a global feeling that Kubernetes log system could be improved but I still need to have a better of that. The global problem to solve is probably the repetition of log, something that's solved by events (or at the way kubectl shows them); so maybe I should move my practice more to events and, when I don't have the answer in the events, file an issue/PR.
the backoff removed the log flooding, right? if so, I think this can be moved to 1.7
c.f.
Thanks You are right, this is not a 1.6 blocker. makes sure the log message is only printed once per un-resolvable ownerReference.

I think the original issue of "spamming error messages" was solved. I created to track the issue of "GC should support non-core API". Closing this issue.


Question: func Stats() (*statsapi.RlimitStats, error) { rlimit := &statsapi.RlimitStats{} <del> if content, err := ioutil.ReadFile("/proc/sys/kernel/pid_max"); err == nil { if maxPid, err := strconv.ParseInt(string(content[:len(content)-1]), 10, 64); err == nil { rlimit.MaxPID = &maxPid </del> <ins> taskMax := int64(-1) // Calculate the mininum of kernel.pid_max and kernel.threads-max as they both specify the // system-wide limit on the number of tasks. for _, file := range []string{"/proc/sys/kernel/pid_max", "/proc/sys/kernel/threads-max"} { if content, err := ioutil.ReadFile(file); err == nil { if limit, err := strconv.ParseInt(string(content[:len(content)-1]), 10, 64); err == nil { if taskMax == -1 || taskMax > limit { taskMax = limit } } </ins> } } <ins> // Both reads did not fail. if taskMax >= 0 { rlimit.MaxPID = &taskMax } </ins> var info syscall.Sysinfo_t syscall.Sysinfo(&info)

PIDPressure is not set to true when the system has insufficient pid resource. PIDPressure is set to true correctly when the system is insufficient in pid resource. See cause below The max pid is not solely controlled by which is referred by . But it is also controlled by (whichever is smaller). However, they have different semantics resulting in the same constraint to how many processes/threads can be running on the system. <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
I asked the same issue before,
My opinion: the minimum of and is the right number to count, for Linux. That will provide better instrumentation. If you're reserving a number of task IDs (PIDs) for system use, that minimum is the number that you're reserving from.
/triage accepted /priority important-longterm
/retitle System maximum PID inconsiderate of sysctl 'threads-max'
BTW I made a thing that might help folks test a fix: Originally I used it to create PIDs, to test whether apps cope with long PIDs. This version keeps the threads around so that you see them in .


Question: // known that the count does not need to be accurate. // BTW, the count only needs to be accurate in a test that // uses FakeEventClock::Run(). <del> klog.V(6).Infof("QS(%s): Context of request %q %#+v %#+v is Done", qs.qCfg.Name, fsName, descr1, descr2) </del> <ins> klog.V(6).Infof("QS(%s): Context of request %q %#+v %#+v is Done", configName, fsName, descr1, descr2) </ins> qs.cancelWait(req) qs.goroutineDoneOrBlocked() }()

Which jobs are flaking: ci-kubernetes-cached-make-test ci-kubernetes-generate-make-test-count10 pull-kubernetes-unit Which test(s) are flaking: TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter Testgrid link: Reason for failure: Data race: Reproduced in ~2% of runs with : Triage board: /sig api-machinery
cc
The test is timing out after 300 seconds, but I couldn't reproduce this over 1000 runs (with and without )
(In case it's not clear from comments: the actual failure is the timeout, not the panic as stated in the OP.)
Finally reproduced the issue locally after many runs:
thank you for reproducing it! are you planning to try to fix it? Thanks!
I tried to, but I couldn't find the root cause. I'll take another look today. /assign
/accept triage
/triage accepted
/milestone v1.21 /priority critical-urgent Raising priority and adding to the milestone for triage since this looks like a data race in non-test code in a default-enabled filter. This unit test did not exist in 1.20 so it's not clear if this is a regression in 1.21 or not. If this issue existed in 1.20, priority can be lowered to important-soon and it does not block 1.21, but should be fixed and picked to 1.20.x and 1.21.x
I'm afraid I won't have the time to look into this in the short term. Unassigning so that someone else can take a look. /unassign
(looking at it now) /assign
found the data race and put out a quick fix for this:
I ran , and it looks good so far:
cherry-picked it for - Can someone please let me know if I did it correctly?


Question: "fmt" "io/ioutil" "os" <ins> "path/filepath" "reflect" </ins> "strings" "testing"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I type <kbdTab</kbd<kbdTab</kbdI would like to see kubectl plugins in the list of suggested subcommands. For example, if I have installed , subcommand completions should suggest . the plugin itself also supports command-line completion, <kbdTab</kbd<kbdTab</kbdshould show suggestions from the plugin command completion. Why is this needed: Plugins fill in niches in the kubectl workflow, so that they are used rather seldomly. This means that plugin users need to memorize plugin names and their options over long times without using them. Therefore, plugin usefulness would be greatly enhanced by suggesting plugin sub-commands and their options.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In fact, already supports command completion. You can run for more information about it. If you are using bash shelljust run , and then it will work!
Hi , but I'm not talking about command completion in general. I'm talking about command completion for plugins. For example, the new plugin system allows to call a plugin at location as . It would be nice to get suggested/completed, when typing <kbdTab</kbd. Further, if plugin names are completed, so should be options for those plugins -- provided the plugin offers some form of command completion.
I try to implement completion for kubectl plugins!
Were you able to make progress on this? I have a working patch for the completion of the plugin name (part 1 of this issue). I can make a PR for it if it can help.
This is a great first step! I do think however, that the real value is auto-completion for the plugin options. I can roughly remember what plugins I have, but not what options they offer. That's the real value here, I think. This will probably need a contract for how plugins make completion available (e.g. by following the kubectl standard of ). WDYT?
I've been thinking about how to do the plugin options completion.

Based on the patch I have now, I believe what will be needed is to be able to ask the plugin what the possible completions can be (based on the user-specified parameters). So the plugin won't need to generate a bash or zsh completion script (which is difficult for the author of the plugin) but just output possible completions. This also has the benefit of being independent of which shell is used. One way is to do as you mention and have the plugin support as a way to ask the plugin what the completions would be. Although it won't behave in the same fashion as (as it won't generate a completion script, and is shell-independent), I like that option because it simply requires a plugin author to provide a different script called which the bash completion logic can call. Another way would be to ask the plugin itself to support a flag such as which would return the possible completions. I don't like this because if the plugin doesn't support this flag, we'd be calling it and who knows what it would do in that unexpected case. The use of makes it easy to check of presence or absence of completion support. So I suggest that the contract with plugins be that a plugin that supports completion would provide two scripts and that the bash completion script call to get the list of completions. Any thoughts?
Hi guys, I've already implemented completion for plugins (subcommands only, doesn't include options) as However it needs to be refactored more. You can try completion for plugins with the following steps: First, prepare the repository, and build a kubectl. Install kubectl-hello plugin for completion testing. Load the kubectl completion code. You can see completion for some plugins.
Cool! You've used the same technique I did. However, may I suggest re-using the code from the plugin package instead of copying it?
Well, you are absolutely right in saying that we can't just call a plugin with some flag to produce the completion script, because who knows what will happen. Therefore, option completion for plugins has to go the route via an external completion script, like However, I'm pretty sure that those completion files need to be bash/zsh scripts, because they need to process contextual information.

For example, say I have a plugin with sub-commands Then I want completion to work as follows: As you say, this will have to be maintained by the plugin authors. Can you take a typical cobra command completion script as basis and work out how kubectl-plugin-completion and this external script need to work together? You could even try to have kubectl as its own plugin, like and or so :grinning:
kubectl already has some dynamic completion code to allow to complete pod names for example, which actually calls kubectl itself. I think I can use the same technique for plugin parameter completion. So here's an example of what I'm thinking: Then when the user types the kubectl completion script will call . This call will return which will be given to the shell by the kubectl completion script. I haven't actually tested this example, so I could be wrong. I'll try implementing it to see if I'm missing something.
Thank you for supporting me ;) I add you as co-author to commit when I send a PR.
Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
I've been working on auto-completion for the Helm project and I have a proposal with a PR that fully supports auto-completion for Helm plugins. I believe the fundamental aspects of the solution would apply just as well to kubectl. Some adaptation would be needed to adjust to how kubectl supports plugins, but I think it wouldn't be very hard.
Is there an ETA on this? If I see it correctly, this is currently preventing plugin authors to implement completions as the generated completion suggestions are based off the root command and not the subcommand aka plugin. I've been trying all day to make this work but failed and that's when I stumbled over this issue..
Several major issues I see here are: We don't want files like as proposed in previous comment. We had a requirement prior to kubectl 1.12 and it didn't look great at all. The simplicity of plugins is valuable to us. (Also, unlike Helm plugins, kubectl plugins don't live at a particular directory, or follow a directory structure.)Plugin completions most likely would be integrated into commands, which a lot of people put in their or file. We need to keep this command fast. So what if the command called a special command on each plugin, in parallel, to inquire if it has completion available (e.g.), with a short deadline, and cached the successful results (maybe keyed with on plugin binary’s stat data, or checksum). That way, cmd could also provide data about plugins, while periodically refreshing the cached completion scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any updates on this one ? is it complete/done?
No update, still open..
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
Sorry to revive this thread a bit but while working on another cli using the same cobra completion system by (implemented by i believe) I found out that it would not be too hard or too disruptive to have a small patch on the shell completion directly. For example with kubectl if we have a patch like this (using zsh but I am guessing the same technique will apply for bash or others) : the patch will check the first command () and the second command for example "krew" exist as and then use it directly for completion otherwise do the normal completion 'thingy'. the only downside is that the initial plugin would not show i.e: if you do you won't get krew as the plugin name, except if we add all plugin to the Root Command like the pull request from does. I don't think it would be too intensive since it is a zsh/bash builtin and in memory. I have made a small asciinema to demonstrate how it works I guess if we want this we may need to do it directly in cobra, do you think a patch like this could get accepted ?

(I had somehow difficulties to get PR reviewer and merged in cobra last time I tried)
Thanks for the suggestion. The idea is good but doesn't generalize perfectly. The command is available for programs that use Cobra but kubectl plugins can be something else (a script, a Python program, a Go program that doesn't use Cobra, etc). So we couldn't do something like this, unless we required every plugin to provide a command. I'm working on porting Helm's plugin completion solution to kubectl and should have something for the community to try soon. Plugins using Cobra will then be able to use your suggestion and get completion super easily.
That make sense I forgot about non go (and non cobra) plugins, thanks I look forward to see how the helm completion extracted will look like !
I've posted something for people to try out in
It would be nice if some plugin developers try out the solution proposed in and provide feedback. since uses Cobra, the solution is very easy. Here is how to try it out: First build kubectl with the PR. Then create your completion script like this: put this new on $PATH. And now test using the newly built (I build mine at ). For completion of flags (which I realize is the real value for the plugin), kubectl needs to first move to Cobra 1.3.
I would be interested in your opinion on the proposed solution. It will not work out-of-the-box with Krew, but I haven't found a way to do better with respect to Krew. You had mentioned this previously, and although it would be a nice solution, I am concerned it would cause problems to any plugin that ignores all arguments and runs no matter what. With such a plugin, calling a command would trigger the plugin to run and possibly perform undesired actions when completions are requested.
I'm sorry about slow responses. Finally had time to read it and parse these. You're right we shouldn't randomly call a plugin binary without knowing it supports completion. From : kubectl already has a mechanism to find all plugins on $PATH, so I agree that at minimum, kubectl<kbdTab</kbdcan and should show plugins installed on the system (even if the plugins themselves don't support shell completion) I think this is a good model.

Just like how kubectl scans $PATH for , it can scan to see if a plugin supports completion. (It's important to define if that binary takes additional positional args or --options) To make the (2) a reality, I support adding an additional field to the Krew plugin manifest. The shape of that would be up for a discussion (I imagine it might be (a) a separate binary that comes in the plugin archive, or (b) allow linking to the plugin binary itself and the plugin reads argv[0] and prints completion, which can be done with (a) anyway). I think it's doable. I'm not very familiar with how shell completion works in practice, when it's called, how lazy can it be loaded, what are performance implications etc, but if you'd like to drive this effort, I think we can discuss a design on Krew side as well. Please bring this up in a SIG CLI meeting to see what the folks think about your proposal.
brilliant! Thanks for taking the time to think about this. Your suggestion to adapt krew sounds great to me. I will continue with the PR on kubectl's side and if it gets accepted we can look at krew.
The as proposed in accepts the exact same arguments as the standard executable. The new completion executable will examine those arguments and will determine what the shell completion choices following them should be.
do you have an insight into whether a plugin developer would develop and ship a separate executable for the completion, or would they simply link the main plugin executable and see if the argv[0] is the kubectl_complete-*? I assume the answer largely also depends on whether the plugin was authored in Go or bash (basically the only two languages I've ever seen a plugin submission in out of 150+).
I think plugins will choose to ship a separate executable. I feel it keeps the two aspects nicely separate (the plugin functionality and the shell completion functionality). Also, it is very easy for a separate executable to call back the main plugin if the developer wishes to do so, so I find that solution the most flexible. For example, for plugins using the Cobra library, the script would call back the executable like this: BTW, I have looked into the 179 plugins in the krew index and I found that 50% (yes, half!)

of them used Cobra (some of them indirectly by calling back ). This means that shell completion should be very easy to implement for all these plugins.
I'm interested in this support and thanks for adding this feature. One question: in which version this is planned to be released? Looking forward to this.


Question: kubetypes "k8s.io/kubernetes/pkg/kubelet/types" ) <ins> var ( // defaultCachePeriod is the default cache period for each cpuUsage. defaultCachePeriod = 10 * time.Minute ) </ins> // criStatsProvider implements the containerStatsProvider interface by getting // the container stats from CRI. type criStatsProvider struct {

Related: On Windows, kubelet reports only the cumulative for cpu usage. On Linux, kubelet reports both and the instantaneous . Although In the future, most components/monitoring pipelines will switch to using the cumulative metrics, a few features still depend on the instantaneous metrics today. Examples are: , HPA/VPA. Kubelet should report this metrics on Windows to support the existing features. /cc who explained to me what features depend on the cpu metrics. /cc /cc
- is this something that's clear enough in the CRI layer that you can implement it, or do we need to fix that first?
/assign
do you have any notes or a doc you can point to that better describes what features depend on which metrics? It would be good for us to share and see if anything other than is broken
The horizontal and vertical pod autoscalers also depend on the
Documentation on metrics API and metrics server (i.e., the aggregator): The metrics-server pod keeps spewing errors like this: Running : I haven't verified whether pod autoscaling works in my cluster, but I assume they won't since metrics server seems to discard the metrics from the windows pods. I believe the implementation is in kubelet (for the docker integration), so this should not need CRI changes.
I think this is generally an issue for non-Docker CRIs (not necessarily Windows-exclusive). The kubelet source code suggests that these metrics will be gathered from the CRI (if the CRI is not Docker). The struct that kubelet expects returned from the ContainerStats RPC does not include usageNanoCores. Does this mean that, until this is resolved, custom CRIs are unable to satisfactorily report metrics to the 'metrics-server'?
- are you able to work on this?
that is a good point. For CRI runtimes, the kubelet should calculate the average over 10s to produce this metric.
Good point. We designed CRI around core metrics API, but some CRI runtimes have integrated with cadvisor directly to fill the gap while we wait for core metrics API. I think it's reasonable for kubelet to do this on behalf of all CRI runtimes. I spoke to offline, and this would probably require no more than 2 days of work. If sig-windows (e.g., can pick this up, that'd be great.

has agreed to review the changes (thanks!) If not, we should clearly document that this is not supported in Windows GA for 1.14.
thanks for the update. we will wait to hear back from
Currently, is got from docker's stats API on Windows. From docker stats API , only is available, hence is not set on Windows: Do you know any way to get instantaneous for Windows containers?
I have no idea. any idea?
I think this is broader than just a Windows issue. Officially supported tools such as the kubernetes metrics-server expect the field, but custom CRIs that conform to the interfaces defined in the package are not given the liberty to return this value.
suggested previously to cache the stats temporarily and compute the stats using the diff. David, could you chime in on how this could be done? Yes, this was mentioned in the previous comments too. There are plans to transition those components off this field, but it's WIP. We are looking for a workaround in kubelet for now.
my biggest worry is that if we say this impacts the horizonal POD autoscaler, how is that passing in our end2end and conformance tests?
Do you test HPA? It's not part of conformance, and I doubt this is included in windows testing.
As to how this could be done, in and we could just add a in the struct, which we assign the previous CpuUsage metric for each container. Then, when we need to calculate the rate, we pass the previous CpuUsage along with the current one to , and calculate a rate based on the timestamp and values.
ListPodCPUAndMemoryStats() we could just add a map[string]*runtimeapi.CpuUsage in the criStatsProvider struct I've done some tinkering on this myself and I originally thought that it would be as simple as that. It gets much more complicated when you have to work in between the existing cAdvisor-based logic. I assume the intent is to cut it out entirely eventually, but the information passed through the CRI doesn't even come close to being so comprehensive. So far I've found that this approach fixes the problem for the "/stats/summary" api, but not necessarily for the other apis. In particular, "/stats/container" seems to be completely based on cAdvisor, being unimpacted by the changes we made to those two functions.

Some monitoring tools and metrics scrapers rely on the "/stats/container" api for their metrics, which means it is insufficient to update one api without doing the other.
We don't need to fill in everything in the summary API, just "usageNanoCores". The "/stats/summary" api is the one used by the metrics server, which is why we care about it in this case. The "/stats/container" isn't something we need to implement to support this.
we a couple of tickets to validate if HPA for Windows containers works. My guess is that it works for memory and not for CPU metrics because of this issue. If this is complicated to fix, i wonder if waiting for the CRI containerD changes in windows would be the best option for supporting this.
containerd and cri-o bypass this issue by leveraging cadvisor as a temporary solution. I don't think that'd work for windows.
Sorry the for late response. HPA is actually working for Windows containers. e.g. in my cluster (v1.12.5), cpu-based HPA is working as expected: Which version of kubernetes are you referring for HPA issues?
are you running heapster, or the metrics server to serve HPA? Heapster uses the cumulative metric, but the metrics-server uses the 10 average rate metric.
metrics-server is used in my cluster.
See .
/milestone v1.14


Question: } volumeObj.globallyMounted = globallyMounted <ins> volumeObj.devicePath = devicePath </ins> asw.attachedVolumes[volumeName] = volumeObj return nil }

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: } if len(awsInstances) == 0 { <del> glog.V(2).Infof("DisksAreAttached will assume no disks are attached to any node on AWS cluster.") </del> <ins> glog.V(2).Infof("DisksAreAttached found no instances matching node names; will assume disks not attached") </ins> return attached, nil } <del> awsInstanceMap := make(map[types.NodeName]*ec2.Instance) </del> <ins> // Note that we check that the volume is attached to the correct node, not that it is attached to _a_ node </ins> for _, awsInstance := range awsInstances { <del> awsInstanceMap[mapInstanceToNodeName(awsInstance)] = awsInstance } </del> <ins> nodeName := mapInstanceToNodeName(awsInstance) </ins> <del> // Note that we check that the volume is attached to the correct node, not that it is attached to _a_ node for nodeName, diskNames := range nodeDisks { awsInstance := awsInstanceMap[nodeName] if awsInstance == nil { // If instance no longer exists, safe to assume volume is not attached. glog.Warningf( "Node %q does not exist. DisksAreAttached will assume disks %v are not attached to it.", nodeName, diskNames) </del> <ins> diskNames := nodeDisks[nodeName] if len(diskNames) == 0 { continue } awsInstanceState := "<nil>" if awsInstance != nil && awsInstance.State != nil { awsInstanceState = aws.StringValue(awsInstance.State.Name) } if awsInstanceState == "terminated" { // Instance is terminated, safe to assume volumes not attached // Note that we keep volumes attached to instances in other states (most notably, stopped) </ins> continue }

In we stopped filtering nodes by their running state. But this means that the NodeName is no longer a unique key. We need to rework the volume status checker to deal with this.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Was discussing this with offline. Why can't we extend the filter to exclude terminated ones? I understand that AWS API doesn't provide a documented way of doing negative filtering, but can we not simply list all the known states we care about? such as - , , etc.
We can do that too :-)


Question: // ServiceAnnotationLoadBalancerInternal is the annotation used on the service // to indicate that we want an internal loadbalancer service. <del> // If the value of ServiceAnnotationLoadBalancerInternal is false, it indicates that we want an external loadbalancer service. Default to true. </del> <ins> // If the value of ServiceAnnotationLoadBalancerInternal is false, it indicates that we want an external loadbalancer service. Default to false. </ins> ServiceAnnotationLoadBalancerInternal = "service.beta.kubernetes.io/openstack-internal-load-balancer" )

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Creating a service with created a load balancer without a floating IP. What you expected to happen: Load balancer should have a floating IP assigned by default. How to reproduce it (as minimally and precisely as possible): Deploy this manifest on a cluster and check the resulting load balancer. No floating IP will be assigned. Anything else we need to know?: The default behavior changed from k8s 1.7.x. It appears that an annotation that will add a floating IP, but the default should not have changed without warning. This also breaks the workflow. For example: Environment: Kubernetes version (use ): v1.8.0-rc.1+coreos.0 Cloud provider or hardware configuration**: openstack OS (e.g. from /etc/os-release): Container Linux by CoreOS 1520.4.0 (Ladybug) Kernel (e.g. ): 4.13.3-coreos Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Oops, default should be external. /assign


Question: // Validate checks that the provided debug options are specified. func (o *DebugOptions) Validate(cmd *cobra.Command) error { <ins> // Attach if o.Attach && o.attachChanged && len(o.Image) == 0 && len(o.Container) == 0 { return fmt.Errorf("you must specify --container or create a new container using --image in order to attach.") } </ins> // CopyTo if len(o.CopyTo) > 0 { if len(o.Image) == 0 && len(o.SetImages) == 0 && len(o.Args) == 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using without specifying should modify existing container images without trying to create a new one. Instead it tries to create a new container and returns an error that isn't set: What you expected to happen: This command should change all container images to instead of returning an error. /sig cli /kind bug /priority important-soon
/assign /triage accepted
My kubernetes cluster version is 1.19.0,and i use kubectl-debug version 0.1.1 try to debug my pod,bug i got error: <img width="1470" alt="image" src="https://user-"Who one know about that


Question: var _ Manager = &manager{} // NewManager creates new cpu manager based on provided policy <del> func NewManager( cpuPolicyName string, reconcilePeriod time.Duration, machineInfo *cadvisorapi.MachineInfo, nodeAllocatableReservation v1.ResourceList, stateFileDirecory string, ) (Manager, error) { </del> <ins> func NewManager(cpuPolicyName string, reconcilePeriod time.Duration, machineInfo *cadvisorapi.MachineInfo, nodeAllocatableReservation v1.ResourceList, stateFileDirecory string) (Manager, error) { </ins> var policy Policy switch policyName(cpuPolicyName) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?


Question: endpointsChanges:      proxy.NewEndpointChangeTracker(hostname, nil, &isIPv6, recorder), syncPeriod:            syncPeriod, minSyncPeriod:         minSyncPeriod, <del> excludeCIDRs:          excludeCIDRs, </del> <ins> excludeCIDRs:          ParseExcludedCIDRs(excludeCIDRStrs), </ins> iptables:              ipt, masqueradeAll:         masqueradeAll, masqueradeMark:        masqueradeMark,

What would you like to be : Change excludeCIDRs from []string to []*IPNet in Proxier and perform the parsing at construction of Proxier. Why is this needed: This was proposed by over comments in Pr This improves the performance of isIPInExcludeCIDRs since the cost of parsing becomes one time.
/sig network /kind cleanup
/remove-kind feature
/triage unresolved I am a bot run by
ping


Question: } } <ins> func TestEphemeralContainerStrategyValidateUpdate(t *testing.T) { test := []struct { name   string newPod *api.Pod oldPod *api.Pod }{ { name: "add ephemeral container to regular pod and expect success", oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, }, } // expect no errors for _, tc := range test { t.Run(tc.name, func(t *testing.T) { if errs := EphemeralContainersStrategy.ValidateUpdate(genericapirequest.NewContext(), tc.newPod, tc.oldPod); len(errs) != 0 { t.Errorf("unexpected error:%v", errs) } }) } test = []struct { name   string newPod *api.Pod oldPod *api.Pod }{ { name: "add ephemeral container to static pod and expect failure", oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", Annotations:     map[string]string{api.MirrorPodAnnotationKey: "someVal"}, }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, NodeName: "example.com", }, }, newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", Annotations:     map[string]string{api.MirrorPodAnnotationKey: "someVal"}, }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, NodeName: "example.com", }, }, }, { name: "remove ephemeral container from regular pod and expect failure", newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, }, { name: "change ephemeral container from regular pod and expect failure", newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image2", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, }, } // expect one error for _, tc := range test { t.Run(tc.name, func(t *testing.T) { errs := EphemeralContainersStrategy.ValidateUpdate(genericapirequest.NewContext(), tc.newPod, tc.oldPod) if len(errs) == 0 { t.Errorf("unexpected success:ephemeral containers are not supported for static pods") } else if len(errs) != 1 { t.Errorf("unexpected errors:expected one error about ephemeral containers are not supported for static pods:got:%v:", errs) } }) } } </ins> func TestPodStrategyValidateUpdate(t *testing.T) { test := []struct { name   string

I created an ephemeral container for a Static Pod using the command. The command succeed. The API server accepted the ephemeral container: However, the container is not created. Its status is also not available: I expected one of these: (a) the API server rejects the request to create an ephemeral container for a Static Pod (b) kubelet creates the ephemeral container, and update the Static Pod's status sub-resource Using a recent version of , v0.17 or newer, and v1.25.0 or newer, create a cluster and create an ephemeral container running . Since the ephemeral container is not running, the output of is 0. The does not address Static Pods. <details</details<detailsNone </details<details</details<details</details<details docker exec -it kind-control-plane containerd --version containerd v1.6.9 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The umbrella issue for ephemeral containers is
/cc
Control plane machines provisioned using run control plane components (etcd, kube-apiserver, kube-controller-manager, kube-scheduler) as Static Pods. Being able to create ephemeral containers for these Static Pods in order to debug the running processes is a use case worthy of mention. If ephemeral containers are, by design, not intended to be used with Static Pods, let's please justify why that is the case, and try to document an alternative.
Thank you for raising this issue. Indeed we either need to update docs or fix it. Can you please provide the kubelet log? We discussed this on triage meeting and not sure if this is actually supported or not. Kubelet log may give us some information quickly in case it is some bug. /triage needs-information /cc
Thanks for triaging it quickly! Yes, I'll do that.

(I didn't do it initially, because even after I bumped kubelet log verbosity to 10, I found no references of the ephemeral container names in the log. I'll just attach the complete kubelet log here.)
Attached is the (compressed) kubelet log running in the kind-control-plane container. I bumped up its log verbosity to 8 prior to creating the ephemeral container. I created the ephemeral container using the command described above at 18:42:31.
hi I was not able to reproduce this. I'm using kind 0.17 and kubectl 1.25.4. Can you try upgrading kubectl? I'd like to know if its some specific version. I looked through the kubelet log and didn't see any signs of trouble. Here's the data from my test run:
Thanks for trying to reproduce this. Your results are consistent with mine. What I am reporting is that, despite having the correct spec, the actual container is not created. However, I failed to explain how to verify that claim. We could verify it, for example, by listing all the processes running in the container. We should find the process, since it would be running in the ephemeral container. In my case, the process is not present: Moreover, since the ephemeral container has a shell executable, I should be able to run . In my case, the command fails: I realize that the is not highly reliable; might not work for other reasons, even if the ephemeral container is created.
I do see a difference in the test results, but I still can't exec into the ephemeral container. I'll run some more tests. Thanks Here is the pod spec showing something for the ephemeral container, but i can't exec into it anyway:
We should not allow ephemeral containers for static pods… the API content is not authoritative for static pods, the manifests on disk are, so I would expect the kubelet to ignore API spec content and the API to disallow adding ephemeral containers to static pods
It sounds like a documentation update would be appropriate here. I think the Static Pods page would be a good place for it. I'll get a PR going. Thanks
/remove-kind bug /kind documentation /sig docs
/remove-triage needs-information Is this good to close?

We could additionally document why you can't add an ephemeral container (which is that mirror pods are essentially read-only).
If the /ephemeralcontainers subresource doesn't currently reject attempts to add to a static pod, I think it should, to make it clearer the addition won't be effective
Does the API server actually know that mirror pods are mirror pods? I wonder how you can tell.
it seems it can if it wants
I think that is just updating this function to fail if the annotation is set, right? ... and add tests, of course can you confirm?
yeah, that's right
any taker on this one?
/kind bug
I'll take it, per the discussion here: add a check in to reject the attempt to add an ephemeral container if the request is aimed at a static pod. Also include a test case. Also some additional documantation comments from this ticket and from kubernetes/website
/assign Thanks


Question: framework.Logf("Reconciled root ca configmap in namespace %q", f.Namespace.Name) }) <del> ginkgo.It("should update a ServiceAccount", func() { </del> <ins> /* Release: v1.26 Testname: ServiceAccount, update a ServiceAccount Description: A ServiceAccount is created which MUST succeed. When updating the ServiceAccount it MUST succeed and the field MUST equal the new value. */ framework.ConformanceIt("should update a ServiceAccount", func() { </ins> saClient := f.ClientSet.CoreV1().ServiceAccounts(f.Namespace.Name) saName := "e2e-sa-" + utilrand.String(5)

[X] Test approval issue: [X] Test PR: 5 Oct 2022 [X] Two weeks soak end date: 19 Oct 2022 [X] Test promotion PR: According to this APIsnoop query, there are still three remaining LimitRange endpoints which are untested. replaceCoreV1NamespacedServiceAccount Using a number of existing e2e test practices a new has been created for one ServiceAccount endpoint. The e2e logs for this test are listed below. This query shows which ServiceAccount endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // (kubecontainer) types. const ( statusRunningPrefix = "Up" <ins> statusCreatedPrefix = "Created" </ins> statusExitedPrefix  = "Exited" )

https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory.

Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.


Question: var map_VolumeError = map[string]string{ "":        "VolumeError captures an error encountered during a volume operation.", "time":    "Time the error was encountered.", <del> "message": "String detailing the error encountered during Attach or Detach operation. This string maybe logged, so it should not contain sensitive information.", </del> <ins> "message": "String detailing the error encountered during Attach or Detach operation. This string may be logged, so it should not contain sensitive information.", </ins> } func (VolumeError) SwaggerDoc() map[string]string {

This came up as a follow up to Also rather than opening 5 different issues - we are going to use this github issue to track all follow up items mentioned here -
/sig storage
/assign /assign
: PR
and the rest is fixed in


Question: } } <ins> func TestDefaultStatefulSetControlBlocksOnTerminating(t *testing.T) { set := newStatefulSet(3) client := fake.NewSimpleClientset(set) informerFactory := informers.NewSharedInformerFactory(client, controller.NoResyncPeriodFunc()) spc := newFakeStatefulPodControl(informerFactory.Core().V1().Pods(), informerFactory.Apps().V1beta1().StatefulSets()) ssc := NewDefaultStatefulSetControl(spc) spc.SetUpdateStatefulPodError(apierrors.NewInternalError(errors.New("API server failed")), 0) stop := make(chan struct{}) defer close(stop) informerFactory.Start(stop) cache.WaitForCacheSync( stop, informerFactory.Apps().V1beta1().StatefulSets().Informer().HasSynced, informerFactory.Core().V1().Pods().Informer().HasSynced, ) if err := scaleUpStatefulSetControl(set, ssc, spc); err != nil { t.Fatalf("Unexpected error: %v", err) } var err error set, err = spc.setsLister.StatefulSets(set.Namespace).Get(set.Name) if err != nil { t.Fatalf("Error getting updated StatefulSet: %v", err) } if set.Status.Replicas != 3 { t.Fatal("Failed to scale StatefulSet to 3 replicas") } // scale the set and add a terminated pod *set.Spec.Replicas = 4 pods, err := spc.addTerminatingPod(set, 2) if err != nil { t.Fatal(err) } if err := ssc.UpdateStatefulSet(set, pods); err != nil { t.Fatal(err) } pods, err = spc.podsLister.List(labels.Everything()) if err != nil { t.Fatalf("Error listing pods: %v", err) } if len(pods) != 3 { t.Fatalf("Expected 3 pods, got %d", len(pods)) } sort.Sort(ascendingOrdinal(pods)) spc.DeleteStatefulPod(set, pods[2]) pods, err = spc.podsLister.List(labels.Everything()) if err != nil { t.Fatalf("Error listing pods: %v", err) } if len(pods) != 2 { t.Fatalf("Expected 3 pods, got %d", len(pods)) } if err := scaleUpStatefulSetControl(set, ssc, spc); err != nil { t.Fatalf("Unexpected error: %v", err) } set, err = spc.setsLister.StatefulSets(set.Namespace).Get(set.Name) if err != nil { t.Fatalf("Error getting updated StatefulSet: %v", err) } if set.Status.Replicas != 4 { t.Fatal("Failed to scale StatefulSet to 3 replicas") } } </ins> func TestDefaultStatefulSetControlUpdateSetStatusFailure(t *testing.T) { set := newStatefulSet(3) client := fake.NewSimpleClientset(set)

/assign flakes in PR jobs more than once per day The can filter down to searching for results for a single test:
This test continues to flake PRs at least once per day:
https://k8s-
looking at it


Question: ] }, { <del> "path": "/api/v1/namespaces/{namespace}/componentstatuses/{name}", </del> <ins> "path": "/api/v1/componentstatuses/{name}", </ins> "description": "API at /api/v1", "operations": [ {

The selflink provided in the api response for component statuses (/api/v1/componentstatuses) is broken. For example: /api/v1/namespaces/componentstatuses/controller-manager (as specified in the response of /api/v1/componentstatuses), will return: { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } while: /api/v1/namespaces/default/componentstatuses/controller-manager, will return the data. In addition, and this is asked for better understanding, are the component statuses namespace scoped? as far as I understood etcd, the scheduler and the controller manager were shared across the kubernetes cluster. If they aren't namespace scoped, what is the logic behind specifing a namespace in these api requests (as shown in ): /api/v1/namespaces/{namespace}/componentstatuses /api/v1/namespaces/{namespace}/componentstatuses/{name} If they are in fact namespace scoped, would you be kind and share an explanation/link to a documentation page?
I agree that it doesn't make a lot of sense for this to be namespaced given what's in it at the moment. Given what I'd like it to contain, the namespace might make more sense in the future. The self link is super broken right now, though, probably due to lack of actual namespace. there's probably a bug in the go-restful setup code for this. Can you take a look at some point? Also it might be worth mentioning that this will probably change in the near future, is working on a replacement.
The whole /componentstatus API is just broken. It needs rewriting.


Question: local file=$2 rm -f ${file} <del> add-to-env ${file} ENV_TIMESTAMP "$(date -uIs)" # Just to track it add-to-env ${file} KUBERNETES_MASTER "${master}" add-to-env ${file} INSTANCE_PREFIX "${INSTANCE_PREFIX}" add-to-env ${file} NODE_INSTANCE_PREFIX "${NODE_INSTANCE_PREFIX}" add-to-env ${file} SERVER_BINARY_TAR_URL "${SERVER_BINARY_TAR_URL}" add-to-env ${file} SALT_TAR_URL "${SALT_TAR_URL}" add-to-env ${file} PORTAL_NET "${PORTAL_NET}" add-to-env ${file} ENABLE_CLUSTER_MONITORING "${ENABLE_CLUSTER_MONITORING:-false}" add-to-env ${file} ENABLE_NODE_MONITORING "${ENABLE_NODE_MONITORING:-false}" add-to-env ${file} ENABLE_CLUSTER_LOGGING "${ENABLE_CLUSTER_LOGGING:-false}" add-to-env ${file} ENABLE_NODE_LOGGING "${ENABLE_NODE_LOGGING:-false}" add-to-env ${file} LOGGING_DESTINATION "${LOGGING_DESTINATION:-}" add-to-env ${file} ELASTICSEARCH_LOGGING_REPLICAS "${ELASTICSEARCH_LOGGING_REPLICAS:-}" add-to-env ${file} ENABLE_CLUSTER_DNS "${ENABLE_CLUSTER_DNS:-false}" add-to-env ${file} DNS_REPLICAS "${DNS_REPLICAS:-}" add-to-env ${file} DNS_SERVER_IP "${DNS_SERVER_IP:-}" add-to-env ${file} DNS_DOMAIN "${DNS_DOMAIN:-}" add-to-env ${file} MASTER_HTPASSWD "${MASTER_HTPASSWD}" </del> <ins> cat >$file <<EOF ENV_TIMESTAMP: $(yaml-quote $(date -uIs)) KUBERNETES_MASTER: $(yaml-quote ${master}) INSTANCE_PREFIX: $(yaml-quote ${INSTANCE_PREFIX}) NODE_INSTANCE_PREFIX: $(yaml-quote ${NODE_INSTANCE_PREFIX}) SERVER_BINARY_TAR_URL: $(yaml-quote ${SERVER_BINARY_TAR_URL}) SALT_TAR_URL: $(yaml-quote ${SALT_TAR_URL}) PORTAL_NET: $(yaml-quote ${PORTAL_NET}) ENABLE_CLUSTER_MONITORING: $(yaml-quote ${ENABLE_CLUSTER_MONITORING:-false}) ENABLE_NODE_MONITORING: $(yaml-quote ${ENABLE_NODE_MONITORING:-false}) ENABLE_CLUSTER_LOGGING: $(yaml-quote ${ENABLE_CLUSTER_LOGGING:-false}) ENABLE_NODE_LOGGING: $(yaml-quote ${ENABLE_NODE_LOGGING:-false}) LOGGING_DESTINATION: $(yaml-quote ${LOGGING_DESTINATION:-}) ELASTICSEARCH_LOGGING_REPLICAS: $(yaml-quote ${ELASTICSEARCH_LOGGING_REPLICAS:-}) ENABLE_CLUSTER_DNS: $(yaml-quote ${ENABLE_CLUSTER_DNS:-false}) DNS_REPLICAS: $(yaml-quote ${DNS_REPLICAS:-}) DNS_SERVER_IP: $(yaml-quote ${DNS_SERVER_IP:-}) DNS_DOMAIN: $(yaml-quote ${DNS_DOMAIN:-}) MASTER_HTPASSWD: $(yaml-quote ${MASTER_HTPASSWD}) EOF </ins> if [[ "${master}" != "true" ]]; then <del> add-to-env ${file} KUBERNETES_MASTER_NAME "${MASTER_NAME}" add-to-env ${file} ZONE "${ZONE}" add-to-env ${file} EXTRA_DOCKER_OPTS "${EXTRA_DOCKER_OPTS}" add-to-env ${file} ENABLE_DOCKER_REGISTRY_CACHE "${ENABLE_DOCKER_REGISTRY_CACHE:-false}" </del> <ins> cat >>$file <<EOF KUBERNETES_MASTER_NAME: $(yaml-quote ${MASTER_NAME}) ZONE: $(yaml-quote ${ZONE}) EXTRA_DOCKER_OPTS: $(yaml-quote ${EXTRA_DOCKER_OPTS}) ENABLE_DOCKER_REGISTRY_CACHE: $(yaml-quote ${ENABLE_DOCKER_REGISTRY_CACHE:-false}) EOF </ins> fi }

I tried to spin up GCE kube environment using kube-up and got the following error: The script would then hang on a subsequent command. The verify-prereqs for kube-up should require yaml support for python and fail early. To rectify, I had to do the following:
Thanks for the report Derek!
Yeah, this broke our Jenkins a while ago, and I thought I fixed it, but didn't. The real fix is in


Question: "//staging/src/k8s.io/apimachinery/pkg/util/version:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", <del> "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", </del> "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/tools/cache:go_default_library", "//test/e2e/common:go_default_library",

Testgrid: https://k8s- https://k8s- Example: https://k8s- This seems to be related to /sig scheduling /kind failing-test /priority important-soon /cc
/assign RaviSantosh Gudimetla
GitHub didn't allow me to assign the following users: RaviSantosh, Gudimetla. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign RaviSantosh Gudimetla Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looked at the kube-scheduler logs for GCE, seems the feature_gate was enabled. I don't see enabled, will setup the cluster locally to test if this is the issue.
have you been able to look into this? These tests are broken right now for unrelated reasons, but once they get fixed I expect to keep seeing the scheduler failure.
/milestone v1.13
is the one open PR expected to fix the remaining failures we are seeing the release dashboards? If not fixed by this week, these failures will block Beta cut targeted for 11/6.
- Apologies for the delay. Yes, this PR should fix the failure, if not, I will disable this test for now and revert the feature promotion PR associated with this test. Also, I need to rethink, if e2e is an appropriate test scenario to handle this feature or should unit tests/integration suffice. I could not find a test which does something like resizing a node in the existing e2e test suite.
I'm looking at this, and I'm not sure this is going to fix the failures. I'm looking at the tests, and it seems to be setting the node status capacity/allocatable to pretend to be a 12 core machine, and then it tries to allocate a pod on every node such that every node is 50% utilized. It thus tries to allocate a pod of ~ 6 cores on the resized node. But that node still only has 2 cores, so it fails to schedule: I think that's why the test has been failing, at least for the past few runs which I checked.

What I have yet to figure out is how this passed here...
is it safe to rephrase this as a pure test design issue then?
the test still failed on the last . Even though it was a couple of hours after PR merged, I dont see it in the delta changes . I will watch for the next run result. Assuming it still doesnt pass (as pointed out about) we need to figure out what this e2e test failure means and should it block 1.13 Beta0 for next Tuesday. Considering the test did pass very few times since its introduction, this seems like an issue with the test design itself and not the feature. Do you have other tests (unit or integration or different approach to e2e) that can infirm us on the health of thew feature? If those exist and are green we can unblock Beta on those results while you determine the right way to author the e2e. Barring those being present right idea, I think we will have to err on the side of caution and roll back the feature itself before Beta. Either way we need to take a call and submit relevant PRs by tomorrow so we get green CI over the weekend, in time for Beta cut on Tuesday.
This is not a blocker, as I told earlier as well, e2e may not be good avenue for this test. I actually thought more about this problem last day. The problem is I updated the node capacity and I was checking if the node could handle the pod based on the scheduler cache which we get from kubelet. But kubelet on the other side is updating it's node information(like CPU, memory etc) in the API server there by causing a race condition. So, it could pass with a very little chance. So, I will remove this test from e2e. Yes, we have unit tests, that help in validating the feature. I will try to get an integration test but this is not a blocker for feature.
Thanks for the explanation and analysis do you plan on adding more e2e tests later or stick to unit and integration for validating this feature? and to weigh in on the path forward i.e., remove e2e tests for this feature and rely on unit and integration.
- We discussed internally in the sig meeting yesterday and it was decided to have an integration test instead of e2e test.

I plan to add an integration test(which we don't have today) later.
since this PR is actually tracking an enhancement work graduating to Beta in 1.13, it should have been tracked via a corresponding issue in k/enhancements repo in 1.13 timeframe. We are now past Feature freeze for 1.13 and its late to open an issue against it. Also we are missing integration tests for this feature, which we are . Given the above two points and from offline discussions with SIG leads regarding the criticality of this feature going to Beta now vs in 1.14 timeframe, I am asking for it to be punted to 1.14. As AIs Please prepare a PR to remove it from Beta in 1.13 Please open a tracking issue in for 1.14 milestone.


Question: // checkerNames returns the names of the checks in the same order as passed in. func checkerNames(checks ...HealthzChecker) []string { <del> if len(checks) > 0 { // accumulate the names of checks for printing them out. checkerNames := make([]string, 0, len(checks)) for _, check := range checks { // quote the Name so we can disambiguate checkerNames = append(checkerNames, fmt.Sprintf("%q", check.Name())) } return checkerNames </del> <ins> // accumulate the names of checks for printing them out. checkerNames := make([]string, 0, len(checks)) for _, check := range checks { checkerNames = append(checkerNames, check.Name()) </ins> } <del> return nil </del> <ins> return checkerNames } // formatQuoted returns a formatted string of the health check names, // preserving the order passed in. func formatQuoted(names ...string) string { quoted := make([]string, 0, len(names)) for _, name := range names { quoted = append(quoted, fmt.Sprintf("%q", name)) } return strings.Join(quoted, ",") </ins> }

What happened: The logic in the checkerNames test for is faulty. Failing the test when the 'wanted' value equals the actual result is the opposite behavior of what we want. What you expected to happen: I expect the test to fail when the desired output is not the same as the actual output. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery /assign


Question: framework.Failf("Error cleanup pod `%s/%s`: %v", metav1.NamespaceSystem, "critical-pod", err) } }() <del> runPausePod(f, pausePodConfig{ </del> <ins> runPausePodWithTimeout(f, pausePodConfig{ </ins> Name:              "critical-pod", Namespace:         metav1.NamespaceSystem, PriorityClassName: scheduling.SystemClusterCritical,

Which jobs are flaking: sig-release-master-blocking Conformance - GCE - master Which test(s) are flaking: E2E: [sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance] Testgrid link: Reason for failure: Anything else we need to know: is currently unavailable links to specific failures in spyglass appreciated From TestGrid alert ...
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I cannot spot any significant logic flaw in the test. A possible cause is the timeout is set too short (1 minute), which may be not enough for the terminating pod to run. Notice that unlike integration test, the time includes the end to end cycle: (1) preemptor being scheduled and evict low-priority pods, (2) low-priority pods being terminating -reported by kubelet that the victims have been deleted, (3) scheduler gets notifications from API server that the victims have been deleted, (4) preemptor gets scheduled, (5) kubelet reports the preemptor is running. /cc for more thoughts.
Maybe, some logs to / would help. Just making the timeout longer don't make sense.
Is addressing this flake?
I think this flake happens before the failure tries to fix:
/kind flake
That's indeed a lot of steps, and flakiness seems very low. I think increasing the timeout for preemption seems reasonable.
is this flake a must-fix in 1.20?


Question: if len(serverList) == 0 { return nil, ErrNotFound <del> } else if len(serverList) > 1 { return nil, ErrMultipleResults </del> } return &serverList[0], nil

Playing around with kind, I noticed the following somewhat unexpected behavior: Using this deployment on a 3-node kind cluster, all two pods get scheduled onto kind-worker node. Now suppose I drain that node by running . The node gets cordon'ed (no more pods can be assigned to it) and then all pods are evicted. All pods are rescheduled to the same node due to the presence of field, but that node has just been marked as unschedulable. It is clear there is a conflict here, and feature takes precedence, so I assume this behavior is expected. Thus, the section could be updated to reflect it is not safe to turn off the node since there are pods running on it. Quoting the doc:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
if you set the field on the node you are on purpose forcing that pod to the node and skipping the You are reasoning is correct but does it make sense to highlight every corner case? anyway, you can always submit a PR to the docs with your suggestions /sig docs /sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details