Question: } // waitForPod watches the given pod until the exitCondition is true <del> func waitForPod(podClient corev1client.PodsGetter, ns, name string, exitCondition watchtools.ConditionFunc) (*corev1.Pod, error) { // TODO: expose the timeout ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), 0*time.Second) </del> <ins> func waitForPod(podClient corev1client.PodsGetter, ns, name string, timeout time.Duration, exitCondition watchtools.ConditionFunc) (*corev1.Pod, error) { ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), timeout) </ins> defer cancel() preconditionFunc := func(store cache.Store) (bool, error) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: test.fit, test.reasons, test.equivalenceHash, <ins> true, </ins> ) value, ok := ecache.algorithmCache[test.nodeName].predicatesCache.Get(test.predicateKey)

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: } func RunHelp(cmd *cobra.Command, args []string) { <del> foundCmd, a, err := cmd.Root().Find(args) </del> <ins> foundCmd, _, err := cmd.Root().Find(args) </ins> // NOTE(andreykurilin): actually, I did not find any cases when foundCmd can be nil, //   but let's make this check since it is included in original code of initHelpCmd

PR broke command due to wrong check . Currently, it displays for every command if you do not specify additional arguments(for example will not work, but will).
please looks at this small fix


Question: ### gRPC Probes Graduate to Beta With Kubernetes 1.24, the [gRPC probes functionality](https://github.com/kubernetes/enhancements/issues/2727) <del> has entered beta and is available by default. You can now [configure startup, liveness, and readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes) for your gRPC app </del> <ins> has entered beta and is available by default. You can now [configure startup, liveness, and readiness probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes) for your gRPC app </ins> natively within Kubernetes, without exposing an HTTP endpoint or using an extra executable. ### Kubelet Credential Provider Graduates to Beta Originally released as Alpha in Kubernetes 1.20, the kubelet's support for <del> [image credential providers](/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/) </del> <ins> [image credential providers](https://kubernetes.io/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/) </ins> has now graduated to Beta. This allows the kubelet to dynamically retrieve credentials for a container image registry using exec plugins, rather than storing credentials on the node's filesystem.

has some broken links. These are relative links that work OK in the release blog article but need to be absolute hyperlinks in the release notes. Links in release notes work. Visit Click on all the links. Specifically, “this guide” in (there are some more, this is one example) No response v1.24.0 n/a n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release and IMO /priority important-soon
Hi Tim, I can help fix this /assign


Question: if err != nil { return input, err } <del> if len(rights) != 1 { </del> <ins> switch { case len(rights) == 0: continue case len(rights) > 1: </ins> return input, fmt.Errorf("can only compare one element at a time") } right = rights[0].Interface()

On , will fail to evaluate a filter when the member does not exist: For instance, if I were to execute Where not all in that namespace had the annotation, this would fail with the error message: This is confusing as it leads me to think more than one item was selected, so while it is not technically incorrect the error message is misleading. I would furthermore expect to more gracefully handle this case -- a missing here means that the filter should fail, not error.
What do you mean by ? How is it different from ?
As in, don't add that item to the list of items on which the following access is made. When I do: And there is no annotation on , I just get nothing back -- no error, just nothing. Similarly, i would have expected the filtered expression to give me the for all of the pods that have an annotation with value . That, anyway is what I understand the JSONPath syntax to mean. When you write You are saying:
cc since you authored and I'm not sure how we want to fix this.
Taking a look now
it looks like you need to do this (note that the annotations portion is wrapped by ): I wrote a unit test and it worked as expected. PTAL!
I'm pretty sure that was a typo on my end but let me check -- do you have a reproducer script?
I see an entirely different error today... weird:
Ok so you have 1 object with annotations, and 1 without. Let me look into that case.
Let me add another annotation to
No difference if both have annotations:
Ok, I'll have to test using kubectl
I'm the Origin bundled but I don't think there are any patches on JSONPath there:
I'm able to reproduce in the unit test when 1 object has annotations and the other does not, if allowMissingKeys is true for the jsonpath parser.
I see. Let me know if I can be of more help.
I think I have a fix. PR coming soon.
Possible


Question: }) } <ins> // UpdateVmssVMWithRetry invokes az.VirtualMachineScaleSetVMsClient.Update with exponential backoff retry func (az *Cloud) UpdateVmssVMWithRetry(resourceGroupName string, VMScaleSetName string, instanceID string, parameters compute.VirtualMachineScaleSetVM) error { return wait.ExponentialBackoff(az.requestBackoff(), func() (bool, error) { ctx, cancel := getContextWithCancel() defer cancel() resp, err := az.VirtualMachineScaleSetVMsClient.Update(ctx, resourceGroupName, VMScaleSetName, instanceID, parameters) klog.V(10).Infof("VirtualMachinesClient.CreateOrUpdate(%s,%s): end", VMScaleSetName, instanceID) return az.processHTTPRetryResponse(nil, "", resp, err) }) } </ins> // isSuccessHTTPResponse determines if the response from an HTTP request suggests success func isSuccessHTTPResponse(resp *http.Response) bool { if resp == nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Current azure cloud provider would fail to detach azure disk when there is server side error, need to add retry mechanism for detach disk operation, while for attach disk operation, it's not necessary since k8s pv-controller will try if first try failed. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.10 ~ master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: return eps, nil } <ins> // GetResponseFromContainer executes a curl via kubectl exec in a container. func (config *NetworkingTestConfig) GetResponseFromContainer(protocol, dialCommand, containerIP, targetIP string, containerHTTPPort, targetPort int) (NetexecDialResponse, error) { ipPort := net.JoinHostPort(containerIP, strconv.Itoa(containerHTTPPort)) cmd := makeCURLDialCommand(ipPort, dialCommand, protocol, targetIP, targetPort) stdout, stderr, err := config.f.ExecShellInPodWithFullOutput(config.TestContainerPod.Name, cmd) if err != nil { // A failure to kubectl exec counts as a try, not a hard fail. // Also note that we will keep failing for maxTries in tests where // we confirm unreachability. framework.Logf("Failed to execute %q: %v, stdout: %q, stderr: %q", cmd, err, stdout, stderr) return NetexecDialResponse{}, err } var output NetexecDialResponse if err := json.Unmarshal([]byte(stdout), &output); err != nil { framework.Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v", cmd, config.HostTestContainerPod.Name, stdout, err) return NetexecDialResponse{}, err } return output, nil } // GetResponseFromTestContainer executes a curl via kubectl exec in a test container. func (config *NetworkingTestConfig) GetResponseFromTestContainer(protocol, dialCommand, targetIP string, targetPort int) (NetexecDialResponse, error) { return config.GetResponseFromContainer(protocol, dialCommand, config.TestContainerPod.Status.PodIP, targetIP, testContainerHTTPPort, targetPort) } </ins> // DialFromNode executes a tcp or udp request based on protocol via kubectl exec // in a test container running with host networking. // - minTries is the minimum number of curl attempts required before declaring

Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I would like to take this /assign
- thanks!

contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!


Question: return nil } <ins> // ResizeController resizes a controller named 'name' by setting replicas to 'replicas' func ResizeController(name string, replicas int, client client.ClientInterface) error { controller, err := client.GetReplicationController(name) if err != nil { return err } controller.DesiredState.Replicas = replicas controllerOut, err := client.UpdateReplicationController(controller) if err != nil { return err } data, err := yaml.Marshal(controllerOut) if err != nil { return err } fmt.Print(string(data)) return nil } </ins> func makePorts(spec string) []api.Port { parts := strings.Split(spec, ",") var result []api.Port

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): HEAD (latest) Environment: dind What happened: Enpoints object is constantly getting updated every 2 seconds, even though nothing has changed in the object. It will produce lot of traffic on any informer watching on endpoints. An update event is produced every 2 second per endpoint. What you expected to happen: Endpoints(any object) should be updated only when there is change in either spec or status. How to reproduce it (as minimally and precisely as possible): Just watch on endpoints Also see the logs from federation controller manager, which is watching on endpoints updates from federated cluster Anything else we need to know: Looking at the endpoints controller code . it looks like reflect.DeepEqual is not working for endpoints labels and Subsets (may be need sorting, just a guess). I also see another issue raised by some time back , related but slightly different. I think this is a bug and if fixed will reduce lot of traffic to etcd.
Those two endpoints specifically are used as shared lock objects by controllers. Agree that's a bad object to use because so many things watch them for updating iptables, etc
code in question:
dupe of


Question: } // We assume docker return us a list of containers in time order if containerStatus, found := statuses[dockerContainerName]; found { <ins> // Populate last termination state if containerStatus.LastTerminationState.Termination == nil { result := self.inspectContainer(value.ID, dockerContainerName, terminationMessagePath) if result.err == nil && result.status.State.Termination != nil { containerStatus.LastTerminationState = result.status.State } } </ins> containerStatus.RestartCount += 1 statuses[dockerContainerName] = containerStatus continue

Currently kubelet reports current status only. If pod's restart policy is not never, the last terminated container status will be override by the latest status, and lost. We should report last terminated container status. I am going to update our v1beta3 api first, but will hold the implementation until we have
did you have a chance to start working on it? IIUC, to support this we need to change the api (), which IIUC from what wrote in some other thread should be done this week. I'm happy to help with it if you want me to help.
Only breaking changes have a deadline. Additive changes can be done at any time.
I am working on this. PR should be ready today or tomorrow.
Also agree with This is not part of API breakage.
Thanks for answers & clarifications and


Question: ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) <del> fp := NewFakeProxier(ipt, ipvs, ipset, nil, []string{"3000::/64", "4000::/64"}) </del> <ins> fp := NewFakeProxier(ipt, ipvs, ipset, nil, ParseExcludedCIDRs([]string{"3000::/64", "4000::/64"})) </ins> fp.nodeIP = net.ParseIP("::1") // All ipvs services that were processed in the latest sync loop.

What would you like to be : Change excludeCIDRs from []string to []*IPNet in Proxier and perform the parsing at construction of Proxier. Why is this needed: This was proposed by over comments in Pr This improves the performance of isIPInExcludeCIDRs since the cost of parsing becomes one time.
/sig network /kind cleanup
/remove-kind feature
/triage unresolved I am a bot run by
ping


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" <del> "k8s.io/apiserver/pkg/authentication/serviceaccount" </del> <ins> "k8s.io/apiserver/pkg/authentication/user" </ins> "k8s.io/client-go/discovery" "k8s.io/client-go/util/cert" apiregistrationv1beta1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1beta1" rbacapi "k8s.io/kubernetes/pkg/apis/rbac" <ins> utilversion "k8s.io/kubernetes/pkg/util/version" </ins> "k8s.io/kubernetes/test/e2e/framework" samplev1alpha1 "k8s.io/sample-apiserver/pkg/apis/wardle/v1alpha1"

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: } }() } <ins> handler := apiserver.TimeoutHandler(m.InsecureHandler, longRunningTimeout) </ins> http := &http.Server{ Addr:           insecureLocation, <del> Handler:        apiserver.RecoverPanics(m.InsecureHandler), ReadTimeout:    ReadWriteTimeout, WriteTimeout:   ReadWriteTimeout, </del> <ins> Handler:        apiserver.RecoverPanics(handler), </ins> MaxHeaderBytes: 1 << 20, } if secureLocation == "" {

I noticed that if I'm following a pod log that is being actively written to it will timeout after 5 minutes. However if the pod is not writing logs the request will not timeout. In that case the request will remain open seemingly indefinitely. I can change the behavior by modifying this line of code: To make it happen faster you can set the value lower. I've been testing with 1 minute. Steps to reproduce: cluster the nginx pod in another terminal generate log output by hitting the pod: (you'll need to change the ip). the logs: In this case whenever the WriteTimeout is reached curl will receive the zero sized chunk and the connection will be closed. If you leave out step 3 curl will continue seemingly indefinitely. This behavior definitely seems odd. Is the usage of WriteTimeout correct here? A quick search shows that others have been confused by the intention of WriteTimeout which is why I ask.
is this same as ?
From discussions on the OpenShift side it sounds somewhat related however what we're talking about is a reasonable (but arbitrary) timeout set. Users of are subjected to this timeout with no good way to pick up where they left off to avoid redownloading potentially large quantities of logs.
To me, it would make sense if the api endpoint had argument which would fetch logs from line (it's easy to track on client on which line we're at), exactly what does: I'm assuming that kubelet gets logs using docker api, unfortunately there is no such thing in docker's api:
I think users that want to do this should get the logs from elastic search instead of from kubectl.
I can't say I disagree about needing something like elastic search in a real world deployment. I guess part of the weirdness is that clients of the logging API rarely know how much is going to be pulled down before they make the request. I guess the answer now if you're following the logs is "5 minutes worth". I know this isn't how it's implemented today but in my mind following the logs should behave more like or . Meaning, when you state you want to follow you don't fetch the entire log file you just get wherever it happens to be (and maybe a little more for context). If you want the whole file you don't pass in .
Any update? This is really annoying and I'm thinking of patching my k8s.

I think the default API timeout has been lengthened to 60 minutes Defined: Used for API server: It looks like the kubelet timeout is still 5 minutes, so that would gate log streaming to 5 minutes:
More nuances in Looks like the API http server times out after 60 minutes: Watches time out between 5 and 10 minutes: Kubelet http server still times out after 5 minutes
Which means that getting logs timeouts after 5 minutes?
yeah
did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
nm, just merged the pr. did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
we could easily bump up the kubelet timeout. The handlers on the kubelet don't have their own built in timeouts (like I did for watch), so a potential downside is long hanging /stats requests could hog connections, and depending on why they're hanging, continue consuming kubelet/cpu/cadvisor time. had a suggestion previously about using elasticsearch, are you running with a cluster that doesn't have the addons (, if you set it up with kube-up it should)?
IMO there's no reason not to make kubelet server's timeout really long, at least for these requests.
I'm trying to put in a fix to increase the kubelet timeouts to 1 hour to match the master. Unfortunately adding proper timeout handling for these cases may not make 1.0, but hopefully this can make it less annoying.


Question: framework.PodNominator parallelizer parallelize.Parallelizer <del> // Indicates that RunFilterPlugins should accumulate all failed statuses and not return // after the first failure. runAllFilters bool </del> } // extensionPoint encapsulates desired and applied set of plugins at a specific extension

Since scheduler Policy API was total removed, we don't support any more. Remove unsupported configuration.
/sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /cc
/sig scheduling (trying to tell the bots to remove )
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /remove-needs sig /remove-label needs-sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's true scheduler doesn't quite use this field anymore, but I'm wondering if some other project like ClusterAutoscaler or simulator still consumes this via .
There's no place we can config this field now by configure file(before we can apply the change by policy api, but now it's removed). So it seems like unless we support this feature with CC, or this field is useless. To other projects, it's the same. When they are using legacy policy api, they can configure this field, but once they synced with the latest kubernetes, with the policy api removed, they can not configure this anymore. In this situation, we should add support in CC. I don't quite understand the relationship between k/k with k/autoscaling or others, I think they sync with k/k periodic.
As I mentioned above, it's still accessible via a framework runtime programmatically, and thus may be still useful. For example, upon a scheduling failure, a ClusterAutoscaler's implementation may choose to dryrun through all filter plugins to ensure a new provisioned node can fit the pod, instead of resolving the failure while failing at another latter filter plugin.
Oh, I haven't thought of configuration via programmatically, that saved my concern. But a little strange.
That's exactly what I meant. Given it's a public option in framework, and there is no alternative if we disabled it, it'd be good to check with potential consumers.

For now I can think of CA ( and k-sigs/scheduler-simulator ( If they can confirm it's not needed in their usecases, I'm fine with removing .
To k/autoscaler, at least these codes are still there together with the legacy policy api. , looking forward to the response.
is not used in sigs/kube-scheduler-simulator.
this isn't used by cluster autoscaler, we can remove it.


Question: ) -prune  ) -name '*.generated.go')) <del> # create a nice clean place to put codecgen there _tmpdir="$(mktemp -d -t codecgen.XXXXXX)" function cleanup { echo "Removing ${_tmpdir}" rm -rf "${_tmpdir}" } trap cleanup EXIT # Sort all files in the dependency order. number=${#generated_files[@]} for (( i=0; i<number; i++ )); do visited[${i}]=false </del> <ins> for generated_file in ${generated_files[@]}; do cat "${generated_file}" > "${generated_file}.original" </ins> done <del> result="" function depends { file=${generated_files[$1]//.generated.go/.go} deps=$(go list -f "{{.Deps}}" ${file} | tr "[" " " | tr "]" " ") candidate=$(readlinkdashf "${generated_files[$2]//.generated.go/.go}") result=false for dep in ${deps}; do if [[ ${candidate} = *${dep} ]]; then result=true fi done echo ${result} } function tsort { visited[$1]=true local j=0 for (( j=0; j<number; j++ )); do if ! ${visited[${j}]}; then if $(depends "$1" ${j}); then tsort $j fi fi done result="${result} $1" } for (( i=0; i<number; i++ )); do if ! ${visited[${i}]}; then tsort ${i} fi done index=(${result}) CODECGEN="${_tmpdir}/codecgen_binary" godep go build -o "${CODECGEN}" github.com/ugorji/go/codec/codecgen </del> <ins> ${KUBE_ROOT}/hack/update-codecgen.sh ret=0 </ins> # Generate files in the dependency order. <del> for current in ${index[@]}; do generated_file=${generated_files[${current}]} initial_dir=${PWD} file=${generated_file//.generated.go/.go} # codecgen work only if invoked from directory where the file # is located. pushd "$(dirname ${file})" > /dev/null base_file=$(basename "${file}") base_generated_file=$(basename "${generated_file}") # temporarily move the generated file to a non-go file so it doesn't influence the verify codecgen mv "${base_generated_file}" "${base_generated_file}.bak" # We use '-d 1234' flag to have a deterministic output every time. # The constant was just randomly chosen. ${CODECGEN} -d 1234 -o "${base_generated_file}.1tmp" "${base_file}" # Add boilerplate at the beginning of the generated file. sed 's/YEAR/2015/' "${initial_dir}/hack/boilerplate/boilerplate.go.txt" > "${base_generated_file}.tmp" cat "${base_generated_file}.1tmp" >> "${base_generated_file}.tmp" rm "${base_generated_file}.1tmp" # restore the generated file mv "${base_generated_file}.bak" "${base_generated_file}" ret=0 diff -Naupr -I 'Auto generated by' "${base_generated_file}" "${base_generated_file}.tmp" || ret=$? if [[ $ret -eq 0 ]]; then </del> <ins> for generated_file in ${generated_files[@]}; do cur=0 diff -Naupr -I 'Auto generated by' "${generated_file}" "${generated_file}.original" || cur=$? if [[ $cur -eq 0 ]]; then </ins> echo "${generated_file} up to date." else <del> echo "${generated_file} is out of date. Please run hack/update-codecgen.sh" exit 1 </del> <ins> echo "${generated_file} was out of date. Please run hack/update-codecgen.sh. (If you're running locally, this was run for you already.)" ret=1 </ins> fi <del> rm "${base_generated_file}.tmp" popd > /dev/null </del> <ins> rm -f "${generated_file}.original" </ins> done <ins> exit $ret </ins>

See, e.g. This seems to be blocking multiple PRs from even being tested.
Looking.
I think I figured it out. The order of generation matters. The verify script subtly breaks that by renaming and checking the files one at a time. The update script deletes everything and then recreates them.
I have a fix. Will send a PR shortly.


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch"

[X] test approval issue : [X] test pr : 28/07/2022 [X] two weeks soak end date : 12/07/2022 [X] test promotion pr : The endpoint as seen on the website is tested but not part of conformance. The APIsnoop query below shows that there is no conformance test for this endpoint. patchCoreV1NamespacedPodStatus Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the pod status endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance /sig node
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale targeting this for 1.25
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ErrReasonNodeConflict ConflictReason = "node(s) had volume node affinity conflict" // ErrReasonNotEnoughSpace is used when a pod cannot start on a node because not enough storage space is available. ErrReasonNotEnoughSpace = "node(s) did not have enough free storage" <ins> // ErrReasonPVNotExist is used when a PVC can't find the bound persistent volumes" ErrReasonPVNotExist = "pvc(s) bound to non-existent pv(s)" </ins> ) // BindingInfo holds a binding between PV and PVC.

What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.

summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point


Question: queueClosed = "scheduling queue is closed" ) <ins> // If the pod stays in unschedulableQ longer than the unschedulableQTimeInterval, // the pod will be moved from unschedulableQ to activeQ. const unschedulableQTimeInterval = 60 * time.Second </ins> // SchedulingQueue is an interface for a queue to store pods waiting to be scheduled. // The interface follows a pattern similar to cache.FIFO and cache.Heap and // makes it easy to use those data structures as a SchedulingQueue.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scheduler places unschedulable pods in "unschedulabe" queue and retries them only when certain events happen that could potentially make them schedulable. This logic works well in almost all scenarios, but inevitable race condition in large distributed systems, could potentially cause some events to be seen before pods are to the unschedulable queue. If this happens, pods may be left in the unschedulable queue and not be retried. Such scenarios should be rare and even if they occur, usually there are other events that trigger a retry and cover them. However, if such scenarios happen in smaller and low churn clusters, other events may not be seen for a while and pods may be stuck in the unschedulable queue for a long time. The scheduling queue () has a method that is triggered every second to move backed off pods to the active queue. We should add another go routine that is triggered every 30 seconds and moves pods whose last schedule time is older than 1 minute from "unschedulble" queue to the active queue. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug I don't know if this should be marked as a bug. I don't have any report of such issue. It is more of a precaution to prevent issues. /sig scheduling
I am worried if this will lead to continuous repeated scheduling failures?
What is the problem with repeated scheduling failures? The scheduler had a FIFO until recently. It used to keep retrying unschedulable pods all the time.
can I take this?
Sure, thanks!
GitHub didn't allow me to assign the following users: denkensk. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It increases the schedule possibility of unschedulble pods and the chances of scheduling failures. Should we also consider latter?


Question: tags = ["automanaged"], ) <ins> # ensure /etc/nsswitch.conf exists so go's resolver respects /etc/hosts container_image( name = "busybox-with-nsswitch", base = "@official_busybox//image", directory = "/etc", files = ["nsswitch.conf"], mode = "0644", ) </ins> # This list should roughly match kube::build::get_docker_wrapped_binaries() # in build/common.sh. DOCKERIZED_BINARIES = { "cloud-controller-manager": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/cloud-controller-manager:cloud-controller-manager", }, "kube-apiserver": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/kube-apiserver:kube-apiserver", }, "kube-controller-manager": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/kube-controller-manager:kube-controller-manager", }, "kube-scheduler": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/kube-scheduler:kube-scheduler", }, "kube-proxy": {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: My broke after , after much debugging we discovered that the API server could not reach itself on , further debugging showed that DNS lookups for localhost were occuring. Long story short, while we have entries for localhost, the api server was not respecting them. What you expected to happen: our standard control plane deployments should not lookup localhost via dns. they should respect /etc/hosts How to reproduce it (as minimally and precisely as possible): create a cluster and monitor network traffic. then try again after adding a minimal to the api server pod. Anything else we need to know?: Prior art: This happens because we build with go's build in DNS resolution, which respects glibc's , but we build images based on busybox for some components (like the apiserver) which does not ship with one. Environment: kubernetes using Kubernetes version (use ): /sig release
/assign cc Thanks a million to and for looking into this with me! this was tricky to resolve.
We should be able to fix this by just adding a minimal to the images make go's resolver happy. alternatively we can standardize on eg which already ships one
oh wow. tough one to debug :)
/milestone v1.13 /priority important-soon
I have a PR under way, will file after lunch!
For now adding the , but discussing with and it sounds like switching to debian-base everywhere is the best long term answer. It will actually save a mb or so on each node :sweat_smile: ... and give a more consistent base for component images to build on.
I just got bit by this.
This issue will bite anyone deploying a statically linked go binary, not just the api server. Can we get kubelet to slide in a default that is sane so it wont keep biting people? Similar to
We'd need a feature request to sig-node and probably sig-network. I do like this idea, but, is used for things other than networking so injecting one might be slightly problematic. In this case it was fine because we control the images and the binaries.

People deploying go images need to make sure they have a sane for now, or not deploy staticly linked binaries, unfortunately.
For 1.13, we should just document the problem
would it be difficult to add the file only if it doesn't exist in the image?
workaround for 1.12: echo hosts: files dns myhostname and add to the ClusterConfig:
Yeah that work around for now, I'm not sure how difficult / expensive it would be for the kubelet to conditionally add this, and it might be a surprising behavior change for arbitrary containers / pods.
it may be surprising, but its very surprising now too, that dns takes precident to hosts file unexpectedly. :( I'd rather it be secure by default and surprise them that its safe. I didn't even know glibc had this backwards behavior when was missing. I haven't run up against a glibc based system that didn't have an nsswitch file. All distro's add one.
hmm... etcd is crashing too.... I hope there is an equiv of etcdExtraVolumes....
never mind on the etcd thing that was something else. The above yaml tweaks did solve the issue.
We have just wasted a TON of time on this when a misconfigured Samsung TV on the other side of the world, plus lack of rigor or our DDNS policy broke our Kube 1.12 clusters. I really don't like that these control plane images are hitting DNS for every lookup of localhost. I am very keen to see implemented.
The control plane images in future releases already contain , which could be cherry picked back. Doing this in an automated fashion like that issue would require bringing a feature to sig-network and sig-node for discussion. I suspect supplying a file only if it doesn't exist in the image would be somewhat expensive compared to what we do currently, and it's definitely a breaking change. In the meantime, if you're containerizing go and not using CGO for networking, you need to add this to your own images. On Tue, Nov 20, 2018, 10:00 nickperry wrote:
I would argue, that 1.12 was a broken release. 1.11 and below did not have the issue.

As K8S is maturing, some of us, especially in more enterprise like environments, are not upgrading as fast, so the issues are being found slower. I think backporting the fix for the control plane images is appropriate to resolve the regression until a more universal fix can be put in place.
that sounds like a great WHODUNIT war story :)
generally I'm not upgrading production workloads particularly quickly either :-) We can file cherry picks back to 1.12, but lack of an in the control plane images is not new: EG does not contain an . IMO, setting on control plane pods was certainly broken, which I questioned, but not just for this reason, and that change was eventually reverted IIRC. /cc Regarding -- because Kubernetes wants to be stable there are processes for proposing and vetting features prior to adding them, particularly for user facing changes! If you'd like to see this, please discuss it with sig-network and sig-node. I don't personally have the bandwidth for this right now and I'm not even sure that it is even the right solution.
yes the change was reverted. it also spawned a little bit of a discussion here for a rather complicated workaround: given this affects images that are already deployed at GCR, it's unlikely that it can be cherry picked. the images do support revisions, but this would also mean porting changes to all the bundled tools that use them as hardcoded tags. also this means revisions for all patch release images. overall i see this as something that will not be approved by sig-network and the GCR maintainers.
I guess to prevent a future regression, we probably need a test that inserts a <some other addressrecord somewhere during the test.


Question: const enableStatefulSetAutoDeletePVC = "[enable StatefulSetAutoDeletePVC]" <del> const enableStatefulSetStartOrdinal = "[enable StatefulSetStartOrdinal]" </del> type testCase struct { name string set  apps.StatefulSet

This applies to StatefulSetStartOrdinal feature gate. Field validation should be done regardless of the feature gate state. From
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig apps


Question: "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/diff" </ins> "k8s.io/apiserver/pkg/admission" admissiontesting "k8s.io/apiserver/pkg/admission/testing" "k8s.io/client-go/informers"

What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!


Question: # etcd - name: "etcd" <del> version: 3.5.3 </del> <ins> version: 3.5.4 </ins> refPaths: - path: cluster/gce/manifests/etcd.manifest match: etcd_docker_tag|etcd_version

Please publish 3.5.4-0 image and update to to benefit from fix for , while still keeping v3.5.3 critical correctness fixes (, ). For recent PRs for update to etcd v3.5.3 see and
/sig release /sig api-machinery /kind bug
/cc /triage accepted
Hey, Can I work on this ? I'm a beginner, but all the good first issues are mostly assigned. Can you guide me on doing this ?
I am working on this. /assign
Hi I would like to work on Make sense to backport in v1.22, v1.23 & v1.24? Thanks!!
hi, it makes sense to me, but requires approval from dependency maintainers. i can only approve the kubeadm changes.
was this backported ?


Question: } } <del> func (config *NetworkingTestConfig) createService(serviceSpec *v1.Service) *v1.Service { </del> <ins> // CreateService creates the provided service in config.Namespace and returns created service func (config *NetworkingTestConfig) CreateService(serviceSpec *v1.Service) *v1.Service { </ins> _, err := config.getServiceClient().Create(context.TODO(), serviceSpec, metav1.CreateOptions{}) framework.ExpectNoError(err, fmt.Sprintf("Failed to create %s service: %v", serviceSpec.Name, err))

What happened: indicates that we're missing an e2e test for. This specific issue is just a tracker to create a new E2E test that demonstrated .
/sig network
/assign
btw, should this be a part of conformance?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I can start working on this.
Sure/ are you working full time on it ? If so I'll drop the issue from my work ... Otherwise we can collaborate as I already have some code for this. Let me know. Was hoping to have this done in the next week or so.
I would love to collaborate, especially if you already have some code. But if you want to drop this from your work, I don't mind taking it on full time either. I can pick up from where ever you left off.
ok, if youve got full bandwidth go for it. if make progress on it this wk ill just drop it . I actually dont know where my old branch is for this :), so i guess youll have to start from scratch. One thing to note ~ Might be seeing if any tests can be reused for this or extended in a simple way by just adding a secondary service, rather then writing a full blown test from scratch. Let me know how it goes thanks !
I will let you know how it progresses. Thanks !
thanks for getting this taken care of :) looks like the PR is LGTM.
Thanks for letting me take a stab at it!


Question: import ( "fmt" "reflect" <del> "strings" </del> "testing" "github.com/google/go-cmp/cmp"

We have a pod fake function in , but this is not a base util anymore, we have a lot of arguments, we can replace this with here /sig scheduling /kind cleanup /help This is an easy one to get start. Cleaner code for maintenance.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? /kind cleanup /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind feature
I want to try this one, the problem we solve here is to replace the *v1.Pod with type PodWrapper {}? /assign
Replace with in
you mean replace ` in I check the code and didn't found out that we have makeBasePod in
I have tried it. But there will be error about cycle import, because package import package, where locate.
yes, I've tried it as well. guess I need to wait for kerthcet's response <img width="805" alt="image" src=""
Sorry, forgot to reply on this, I introduced another PR to help solve this import cycle, but waits for other maintainers' response. So generally, if that pr accepted, we can work on this one then. Thanks for the volunteer.
the import cycle is resolved, you can start your work now, thanks!
thanks for the ping!


Question: return options } <ins> // appendOptions appends options to the given list, but does not add duplicates. // append option will overwrite the previous one either in new line or in the same line. func appendOptions(options []string, newOption ...string) []string { var optionMap = make(map[string]string) for _, option := range options { optName := strings.Split(option, ":")[0] optionMap[optName] = option } for _, option := range newOption { optName := strings.Split(option, ":")[0] optionMap[optName] = option } options = []string{} for _, v := range optionMap { options = append(options, v) } return options } </ins> // appendDNSConfig appends DNS servers, search paths and options given by // PodDNSConfig to the existing DNS config. Duplicated entries will be merged. // This assumes existingDNSConfig and dnsConfig are not nil.

If the content of is only could recognize the last option and is ignored. And user has to use to workaround the issue. But this format is valid actually and able to read correctly in golang and libc. Able to parse multiple options lines in correctly. Use in the host's file and then inside POD's , it would become No response Any version. N.A. Ubuntu 22.04 No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-node </details
/sig node
Seems this behaviour is introduced by you at , is it an expected behaviour or a bug?
are those options parsed and used correctly or one of both is ignored?
Others are ignored and online the last line is parsed by kubelet correctly.
and in a normal , does libc use all the options or ignore some and use only one?
I tried centos/ubuntu, both will append extra options to the options list and overwrite them if it already exists by the later settings. For instance, is the same as . But I still think the workaround looks a perfered format.
Yes, but it looks like a bug since kubelet's behavior is different.
I tried to fix it in But I am still not sure if this is a valid fix. I will wait for pr review comments or further confirmation about whether this is valid.
/sig network
EDITED The feature was by a long time ago by There is a design kubernetes/community. The PR does not follow the design per The KEP can be found in KEP
/remove-sig node
Previous discussion and interesting comment from
Just want to clarify that these two are not related --- the change does not follow the design.
Just have the book on my desktop. !
I think the book just said could write it as , but didn't mention is invalid.

And since glibc/golang DNS resolve is able to recognize it correctly, should we consider it as a de-facto way that we should support?
what about muslc, does it support it? we have to support muslc too, a lot of busybox/alpine users can break
I did some initial test with busybox:1.34.1-musl. It supports multi options. for , it works with multi-options. ndots:1, ping will fail for .
/assign
I'm trying to understand this, how the options are merged if they are duplicated? the man page for is clear ,notation%20as%20per%20RFC%. for only says it seems that apply the same logic to handle conflict if I try with this only the last option is used (same for muslc and libc) if I have two different options in different lines then things start to get more complicated, because I can not test all possible options combinations and musl resolver is not compatible with glibc resolver , see I'm worried we can break musl based containers by assuming the host configuration, that is mostly libc based, will be compatible with the container. I think we should be conservative and document this behavior of the dns options, and ask users to merge the options in the same line, raise a warning of something too EDIT I see is suggesting to parse all options and merge them in one line :eyes:
Add a small warning in log maybe another option. We may document it with some details. This is a little aggressive way.
I still can't find any spec that says how this is SUPPOSED to work, and I share concerns that this is likely to break someone somewhere. Did someone encounter this in the wild? Does it REALLY make sense to copy ALL of the options from the host config? Does it make sense to copy ANY of them? Looking at the man page, they seem very specific and maybe we should just leave them alone. What does Docker do here?
I found that there is a logic to merging DNS options in the pod DNS spec generating process. So I followed the logic. If the pod is using with , the current kubelet will merge the node's last option line in with . I did a simple test in Centos 7.9 with docker 19.03.8 like below.

keeps the options as is if no new DNS options. will overwrite the local DNS options. there are multi , docker will not merge them and keep them in one line.
Absent a real user who is really impacted by this, my inclination is to (again) do nothing. The risk of breakage is non-zero. The reward is zero (nobody is complaining, IIUC).
Hi, actually I am impacted by this (in production services) . And looks like I am not the first user who raise this, as other issues link to this thread.
Thanks for your input, I checked muslc and verified via code, muslc would parse all options if we write multiple line . So I don't think would break any user. Instead, it addresses the similar problem happens to muslc users as well.
OK. I am accepting it. Since libc's seem to do this anyway, I don't see why we should break it.
Many thanks


When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> //go:build linux // +build linux /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import "os" // chown changes the numeric uid and gid of the named file. func (w *AtomicWriter) chown(name string, uid, gid int) error { return os.Chown(name, uid, gid) } </ins>

Install a cluster with v1.29.4 with linux node as master and worker node as windows kubeadm upgrade apply v1.30.0 fails with --ignore-preflight-errors` Describe Pod events: It seems that previously the pod was Pending as well, but this was ignored, because the jobs was successfully deleted in defer and return value was overridden with nil. Upgrade health check job expected to be scheduled only on Linux nodes using node selectors, rather than on Windows nodes. See What happened? No response_ <detailskubeadm version (use kubeadm version): v1.30.0 Environment: Kubernetes version (use kubectl version): v1.30.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 22.04.2 LTS, Windows Server 2022 Datacenter Kernel (e.g. uname -a): 5.19.0-1025-aws Container networking plugin (CNI): calico </details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in 1.30.1 /close
Closing this issue. <detailsIn response to : kubeadm version (use kubeadm version): v1.30.0 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubeadm
we are seeing this issue for 1.30.1 version too with upgrade health check pods getting scheduled on windows worker nodes.
looks like I was mistaken and this issue is different. but the health check uses the pause image, which is multi os, supposedly. the error is strange. can you show the full log output. we might have to apply a node selector on the job spec and backport to 1.31 another bug fix.

I'm not sure if we should add a node selector on job spec. the job uses the pause image to create a container. It doesn't have specific requirements for the node. After I look at the projected volume plugin, I'm confused why the above event occurs. On windows nodes, the function is empty. It doesn't do anything. On linux nodes, the function will set the owner of the volume. FYI: If a log file from the kubelet is provided, it will be helpful to understand the issue.
thanks for checking I also think there should not be an error. maybe we are hitting a core kubelet bug. is the kubelet on the windows node also 1.30?
please ignore the above comment, it's incorrect. I may find the root cause. It's a bug in kubelet when kubelet is running on Windows and the pod configures the SecurityContext with a RunAsUser.
/sig storage /sig windows
/assign I'm trying to fix the kubelet bug.
so this is not a kubeadm bug and we can move it back to the k/k repo?
/transfer kubernetes
/retitle kubelet on Windows fails if a pod has SecurityContext with RunAsUser.
Because of historical reasons, the and in is only designed for the Linux, not Windows. See for more details. adds a new field to the PodSpec, which can be used to identify the OS of the pod. It's an optional field and no default value. If unset, the above fields are not restricted by the api-server. In other words, the following job is allowed to be created. But it will fails to start on Windows nodes. And the root cause is that the is called by the . To fix this issue, the AtomicWriter can check the current OS of the node and skip the if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. I'm not sure if the failing job is expected or not. If it's expected, kubeadm has to change the job manifest to make it work on only Linux nodes. Why I ask this question is that I saw , which is used to set the ownership of the volume, if the os is Windows, it will become a no-op.

/cc
/cc
/triage accepted
slack discussion is happening here: but we have agreed that this is a kubelet problem. on the kubeadm side you can use the --ignore-preflight-errors=CreateJob option to skip the check. unclear when or if a kubelet fix will be and to what versions it will be available.
To fix this issue, the AtomicWriter can check the current OS of the node and skip the os.Chown if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. please see: do you want to upstream your carry?
thank you. I raise a PR and add as a co-author. I add an e2e test. do you know which job can test windows?
is the e2e test job for Windows


Question: } func parseSelectorQueryParams(query url.Values, version, apiResource string) (label, field labels.Selector, err error) { <del> label, err = labels.Parse(query.Get("labels")) </del> <ins> labelString := query.Get("labels") label, err = labels.Parse(labelString) </ins> if err != nil { <del> return nil, nil, err </del> <ins> return nil, nil, errors.NewBadRequest(fmt.Sprintf("The 'labels' selector parameter (%s) could not be parsed: %v", labelString, err)) </ins> } convertToInternalVersionFunc := func(label, value string) (newLabel, newValue string, err error) { return api.Scheme.ConvertFieldLabel(version, apiResource, label, value) } <del> field, err = labels.ParseAndTransformSelector(query.Get("fields"), convertToInternalVersionFunc) </del> <ins> fieldString := query.Get("fields") field, err = labels.ParseAndTransformSelector(fieldString, convertToInternalVersionFunc) </ins> if err != nil { <del> return nil, nil, err </del> <ins> return nil, nil, errors.NewBadRequest(fmt.Sprintf("The 'fields' selector parameter (%s) could not be parsed: %v", fieldString, err)) </ins> } return label, field, nil }

While developing a test, I POSTed and incorrect /events and got a 500. Should have been some other error code, I think? Action: POST to /api/v1beta1/events with body: Response was 500 with this "kind":"Status","creationTimestamp":null,"apiVersion":"v1beta1","status":"Failure","message":"couldn't get version/kind: YAML error: line 12: did not find expected ',' or '}'","code":500} Note there is one missing close brace. Error message in body is quite helpful. But should the error code be 400 or
/cc


Question: case 0: return nil, nil, errors.NewBadRequest(fmt.Sprintf("a container name must be specified for pod %s", name)) default: <del> containerNames := getContainerNames(pod) return nil, nil, errors.NewBadRequest(fmt.Sprintf("a container name must be specified for pod %s, choose one of: [%s]", name, containerNames)) </del> <ins> containerNames := getContainerNames(pod.Spec.Containers) initContainerNames := getContainerNames(pod.Spec.InitContainers) err := fmt.Sprintf("a container name must be specified for pod %s, choose one of: [%s]", name, containerNames) if len(initContainerNames) > 0 { err += fmt.Sprintf(" or one of the init containers: [%s]", initContainerNames) } return nil, nil, errors.NewBadRequest(err) </ins> } } else { if !podHasContainerWithName(pod, container) {

Im extracting logs part from The current behavior is as follow: when pod has one container return logs for this one container when pod has more than one container returns error: Error from server: a container name must be specified for pod kube-dns-v14-z0ho6, choose one of: [kubedns dnsmasq healthz] As you can see in both cases init containers are ignored. The question are: in first case should we return error and list all containers including init containers? in second case should we also list init containers?
first case is designed on purpose that way for the usability, especially the users who switch from docker single container to Kubernetes pod. But what you suggested above for the second case are great, we should do it. Mind to send pr fixing it? Thanks!
sure, I will prepare pr. I'm wondering if we should somehow distinguish init containers and normal containers. we can add to init container name but this may be misleading. we can also add second list with init containers. Something like these: Or just show all in one list without any changes. What do you think?
I prefer not introducing a prefix "init:" here. I like the second option better. cc/ WDYT?
I like the second option as well. We generally avoid prefixes or special syntax except in rare cases.
since I know you are also looking at code in these places.
+1 for the second option.


Question: stop() // Don't defer this. See os.Exit documentation. checkNumberOfGoroutines := func() (bool, error) { <del> // Leave some room for goroutines we can not get rid of // like k8s.io/klog/v2.(*loggingT).flushDaemon() // TODO(#108483): Figure out if we can reduce this //   further (ideally down to zero). if dg := runtime.NumGoroutine() - before; dg <= 4 { </del> <ins> // We leave some room for leaked goroutines as there are // still some leaks, mostly: // - leak from lumberjack package we're vendoring // - leak from apiserve healthz // - leak from opencensus library // Once fixed, we should be able to bring it down to zero. if dg := runtime.NumGoroutine() - before; dg <= 3 { </ins> return true, nil } // Allow goroutines to schedule and die off.

The integration framework runs an apiserver and sometimes other components before each test execution, however, the shutdown is not clean and is leaving a lot of goroutines orphan that keep accumulation between different runs. Some orphan goroutines are connections to etcd, and since the etcd process is shared for all test, we don't know if this can cause contention and flakiness on the job. We are logging the number of goroutines leaked per test binary execution, it is important if we can reduce them and more important we don't regress and this number keeps growing. This issue will record the state as today for reference:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
hmm, not doing better
I'll take the scheduler part for a deep investigate.
For people that want to get access to the stack traces, just apply this patch, so the framework will dump the output of the goroutines after each execution: [x] Apiserver was shutting down the listener before the hooks could finish, hence the controllers couldn't shutdown gracefully since there was no http server [ ] The storage cacher doesn't differentiate between not read and stopping - Missing integration test reproducing the issue, only reproduced with unit tests [ ] admission plugins can't shutdown gracefully - - there is a PR addressing However, most of the contributions to goroutines leaks are grpc connections that are related to the storage cleanup, there is also a TODO in the code and an open issue I've also tried to reduce the number of connections between storage and etcd, right now, each Storage creates an independent conection to etcd, but sharing the etcd client doesn't work
I think I made some progress with it. I was playing with ttlcontroller test (as it's fairly simple one for the playground and I'm familiar with it, but symptoms in many tests are exactly the same) In I actually managed to reduce amount of leaked goroutines from 1023 to 85.

It's not mergable in this state, but it clearly shows that destroy functions for storage are not called in integration tests. I will try to dig deeper into those and clean this up - but I think the above clearly shows where the problem is. - FYI ^^
is out for review and it's making most of the job here There are still leftovers, but those are mostly oneof's here and there - we should address them independently.
And a bit more longerterm cleaner approach is in
With we're down to 14. I admit that this is one of the simplest tests, but apiserver shutdown is pretty clean once that PR merges.
/honk
! <detailsIn response to : /honk Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
a leftover for the storage cleanup in
Re comment above - yeah, I'm aware, we need to clean it up. FTR:
awesome, we are several order of magnitud better
Looking further into it, in different tests I'm seeing a lot of goroutines with the top of the stacktrace like this: should hopefully address those once it merges... ^^
OK - I think I have now the biggest thing contributing factor to the remaining tests, although I don't yet fully understand it and don't know how exactly to fix it. The main problem is basically that TestServer is not finishing its shutdown: So what exactly is happening is: we run TryDown the TestServer there: this underneath is triggering run of PreShutdown hooks: in the meantime ShutdownInitated signal is signaled: as part of running preshutdown hooks, the bootstrap controller is stopping endpoint reconciler: and then as part of doReconcile, the GET requests hangs forever: By adding time.Sleep(5s) (didn't try smaller values TBH) before the ShutdownInitiated is signalled, the TestServer is shutting clearly: Obviously that's not a solution, but I'm wondering why the GET request mentioned above is hanging forever when ShutdownInitiate is signaled and what we can do with this. - thoughts? [The easiest repro is to run: Adding this sleep basically reduces the number of leaked routines from O(1000) to O(20).]

<--- UPDATE ---In fact, setting ShutdownDelayDuration to 5s has the same effect. So there are two main questions: if setting ShutdownDelayDuration is reasonable why the request actually hangs forever instead of finishing with some error... <--- UPDATE ---I think there must have been something different, as I'm no longer able to reproduce the improvement with my current code. Will try to dig deeper. <-- UPDATE ---OK - the last commit in brings it back to fixed.
I thought that the reconciler hangs because of
Not in this place - here the usual GET request hangs - I will keep looking over the next days.
OK - I understand now and the above isn't really true. The request wasn't really hanging forever. What was happening is that it was being rejected by the WaitGroup handler: and retried by the client-go internals... So I guess from that perspective setting non-zero ShutdownDelayDuration actually makes sense.. - thoughts? ^^
mhm, not sure why setting should help. In general, the server shout stop listening and the HTTP std library should instruct the client's stack to tear down the underlying TCP connection. In that case, the client-go should fail on dialing and stop retrying releasing the server. thoughts?
It's that with ShutdownDelay, the "stopCh" is signaling PreShutdown hooks (that still want to do some work, including API calls) and if we immediately start rejecting that, they will not be able to do that work...
Discussed a bit more offline - I'm going to investigate blocking closing the waitGroup filter on preshutdown hooks. Will try to send a PR in upcoming days.
We're getting there - this is where we are now: I have a bit more in my local branch to upstream, but I need to clean them up and may not have time in the next week or so.
where are we standing today?
Getting close:
A bit more coming soon.
This is where we are now: I understand some of them (though fixing is potentially non-trivial) and need to understand the others (in particular serving). And we should prevent regressions by failing if we have more leaking goroutines - will be looking at that over upcoming weeks.

OK - so here is where we are (with the current check set to not leaking more than 4 goroutines): Comparing to the first comment in this issue - that looks like a significant improvement :) Here is the summary of I think the remaining goroutines in the above files: [x] integration/apiserver/tracing: The goroutines are coming from otel exporter (in particular grpc layer). It seems that something is not closed, although required digging deeper into otel exporter. [ ] integration/serving: The goroutines are coming from the failure case: And they basically are coming from the fact that Queue when created is spawning a goroutine: which is only shutdown on Shutdown() call. But Shutdown in all controllers is only called in defer of controllers of the function that is actually not being run in failure cases. The actual queue that is problematic here is coming from here: [x] testing/integration/controlplane/transformation The goroutines are coming from the fact that we're not closing the kms transformer (grpc goroutines) that is being created. We basically create a grpc service and never close that: Fixing would probably require changing transformer interfact to implement Close method or sth like that.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
- can you please help with the apiserver/tracing test mentioned in
looking
Opened to try fixing the goroutine leak.
It seems the fix worked fine - I'm marking the tracing as fixed - thanks I'm also trying to fix couple other smaller things.
I've opened as a final step. With that PR we're in good enough shape that I'm going to close this issue as further improvements would have negligible impact.


Question: // may make pending pods with matching affinity terms schedulable. func (p *PriorityQueue) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) { p.lock.Lock() <del> p.movePodsToActiveOrBackoffQueue(logger, p.getUnschedulablePodsWithMatchingAffinityTerm(logger, pod), AssignedPodAdd, nil, pod) </del> <ins> // Pre-filter Pods to move by getUnschedulablePodsWithCrossTopologyTerm // because Pod related events shouldn't make Pods that rejected by single-node scheduling requirement schedulable. p.movePodsToActiveOrBackoffQueue(logger, p.getUnschedulablePodsWithCrossTopologyTerm(logger, pod), AssignedPodAdd, nil, pod) </ins> p.lock.Unlock() }

/sig scheduling /kind cleanup Ref: A test helper function creates a goroutine to pop a Pod from the scheduling queue. This goroutine never stops until the scheduling queue is ed or is somehow stopped for waiting for a new Pod. Ideally, when reaching timeout, we should somehow stop from waiting for Pod to be enqueued to activeQ.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We can replace this function with and it should be enough? I think we no longer need to maintain this function.
Not enough, the core of this problem is that we cannot cancel once we start . Even if we change function to whatever, we will hit this problem. So, we need:
Or, alternatively and much more easily, maybe we can just replace all with and remove . kills the entire test if reaching timeout and we could care less about leaked goroutine then.
I don't think we can do this. From the test case: The expected behavior of this case is that we receive a timeout to make sure no pods are to . So, reaching timeout is acutally a success. So, I think: option 1: using sth like this in : option2: do sth in the test code This part is where it confused me. The current code is fine as we closed schdulerQueue after called , and after I think there is no test case using this this function anymore(seeems that you refactored the test), so are we trying to solve a potential problem?
Oh, then we can just remove NextPod in the PR.
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

/sig scheduling /kind bug AssignedPodAdded(): AssignedPodUpdated(): When assigned Pods are created/updated, cluster events (Pod/Add or Pod/Update) should be delivered for all plugins that subscribe those events. But, actually pre-filters Pods to receive those events based on Pods' affinity and . - meaning only PodAffinity/NodeResourceFit receive them actually. The problem here is similar to - in in-tree plugins, PodAffinity, PodTopologySpread and NodeResourceFit subscribe Pod add and/or update events. And, Pods rejected by PodTopologySpread will never be requeued to activeQ by Pod-related events. The same happens for custom plugins registering pod add and/or update events. All Pods will get chances to get back to activeQ by Pod-related events. Run ↓ in No response <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: features:   feature.DefaultFeatureGate, setupFunc: func(t *testing.T) { t.Helper() <del> featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, features.DisableKubeletCSRAdmissionValidation, true) </del> <ins> featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, features.AllowInsecureKubeletCertificateSigningRequests, true) </ins> }, }, {

This comes from We've a feature gate, . However, we try to name feature gates after the positive thing they add, even if it's on by default, or even if we're setting if false because the positive thing is being dropped (rare, but it happens). Typically we don't start feature gate names with , as it makes the docs jar: Example quote 1: To temporarily revert to the legacy behavior, enable the feature gate. vs Example quote 2: You can enable the feature gate if you run code that relies on the previous behavior. You should plan to migrate. and because disabling things isn't really a feature. Some examples of gates you can or once could disable by turning them off_ include: - - - So, I recommend a rename of the gate before the v1.31 release. /sig auth architecture
FYI
Example gate names: (off by default) (off by default) (on by default) (on by default) (on by default) from (off by default, deprecated)
/milestone v1.31
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.31 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another possibility: , deprecated, off by default
Hello there! I am new to kubernetes and I want to contribute in it. Is it something that I can contribute to?
This issue is not a good issue for a first-time contributor to work on.
I'm a fan of , deprecated, off by default
hoping one of you are doing the PR? :) /milestone v1.31


Question: userInfo, _ := request.UserFrom(ctx) staticAdmissionAttributes := admission.NewAttributesRecord(nil, nil, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Update, userInfo) admissionCheck := func(updatedObject runtime.Object, currentObject runtime.Object) error { <ins> // if we allow create-on-patch, we have this TODO: call the mutating admission chain with the CREATE verb instead of UPDATE </ins> if mutatingAdmission, ok := admit.(admission.MutationInterface); ok && admit.Handles(admission.Update) { return mutatingAdmission.Admit(admission.NewAttributesRecord(updatedObject, currentObject, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Update, userInfo)) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.


Question: importpath = "k8s.io/kubernetes/pkg/scheduler/framework/plugins/podtopologyspread", visibility = ["//visibility:public"], deps = [ <del> "//pkg/scheduler/algorithm/predicates:go_default_library", </del> "//pkg/scheduler/framework/plugins/helper:go_default_library", <del> "//pkg/scheduler/framework/plugins/migration:go_default_library", </del> "//pkg/scheduler/framework/v1alpha1:go_default_library", "//pkg/scheduler/listers:go_default_library", "//pkg/scheduler/nodeinfo:go_default_library",

Part of /priority important-soon See example /sig scheduling /kind cleanup
/help
/assign
/remove-help