Question: scope.name = selectorName } <ins> // for request like '/api/v1/watch/namespaces/*', set scope.namespace to empty. // namespaces don't populate metadata.namespace in ObjFields. if c.groupResource == coreNamespaceResource && len(scope.namespace) > 0 && scope.namespace == scope.name { scope.namespace = "" } </ins> triggerValue, triggerSupported := "", false if c.indexedTrigger != nil { for _, field := range pred.IndexFields {

We upgraded our test cluster to (GKE ) and noticed that some of our tests began to time out when watching for events that never arrived. Upon investigation, it appears that there is a regression in the deprecated path-based watch API for at least e.g. . We have not been able to reproduce this with any other resource type. When creating a watch for a namespace (see ) we see the initial event, but then we do not see subsequent events if we modify the namespace e.g. add a label or annotation. I would expect to see events when the namespace is altered. E.g. adding a label should produce a event with the contents. a new test namespace e.g. a proxy to the master the watch path of the namespace that has been created e.g. another shell, modify the namespace to add a label to it. that there is no modified event in the watch. It is understood that the based paths are and have been for some time deprecated, we are working on moving to use the watch query param instead. <detailsWe are using Google Kubernetes Engine, please see version above. </detailsNo response No response No response No response
/sig api-machinery
The addWatcher add watch to allWatchers with with the value namespace not empty. But the function dispatch event with an empty namespace.
This issue is limited to watchers of the core namespace resource using the deprecated endpoint and watching a single namespace. This was introduced in 1.27 in (which was merged with test coverage in 1.27 in )
Hi, just to confirm this was working fine (and still is on our production clusters) on 1.28.7 it's only since 1.28.8 that we can see this bug, so slightly confused by your comment (and issue rename) to say 1.27+?
I reproduced the break at HEAD of release-1.27, and assumed the break came in with , but I was wrong.

release-1.26 HEAD: works v1.27.0: works release-1.27 HEAD: broken v1.28.0: works v1.28.7: works per your feedback v1.28.8: breaks per your feedback release-1.28 HEAD: broken That means that watch cache changes cherry picked back to 1.27 and 1.28 regressed something, and I don't know exactly what the breaking change was. The only thing that looks at all relevant between 1.28.7 and 1.28.8 is , but I don't see how that would only impact the single-item watch cc
Did you send the request with a resource version, or any other parameter? If the parameter is not empty, will work, or the old watcher will work. This makes a big difference.
This commit add the feature gate .
No, I just sent and that broke somewhere between 1.27.0 and
If you are asking me, I did not send any additional query parameter, no. On our cluster: works as expected, we see events when we add labels etc. On our cluster: does not work, we do not see events.
Ok, figured it out... 27.0 which is what made the watch cache not handle correctly (merged in with test coverage) also which routed watch requests without a resourceVersion parameter directly to etcd, bypassing watch cache 27.13 / 1.28.9 / 1.29.4 / 1.30.0 moved the behavior from behind an off-by-default gate because it was a massive performance regression. This had the side effect of making watch requests without a resourceVersion parameter get handled by the watch cache, and get exposed to this issue.

(GKE accelerated that specific fix internally to resolve that performance regression, so it showed up in a GKE 1.28.8-gke.X build, which is what confused me when looking for the delta between v1.28.7 and v1.28.8) in [1.27.0-1.12.12], [1.28.0-1.28.8], [1.29-1.29.3]: routes to etcd and works (because of ) routes to etcd and works (because of ) routes to watch cache and fails (where X is not 0) routes to watch cache and failsin 1.27.13+, 1.28.9+, 1.29.4+, 1.30.0+: routes to watch cache and fails routes to watch cache and fails routes to watch cache and fails (where X is not 0) routes to watch cache and fails
The workaround for anyone encountering this issue is to use the non-deprecated watch endpoint (and is a generally good change to make anyway) The Kubernetes fix for this I would recommend to restore pre-1.27 behavior is in my comment in Once that is done and test coverage , I'd be happy to see that backported to 1.27+
/cc
GitHub didn't allow me to assign the following users: tinatingyu. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: yt2985. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
just to verify, are you planning to take to completion?

if so, it would be ideal to complete that so the 1.27 backport can happen before 1.27 reaches EOL in June
I saw I was assigned with this issue. Taking a look...
sorry, was tagging you in since you helped bring in and might have context, but realized I should check with first since they have a fix in progress
Yew, working on it and writing test case.
cc
is this issue resolved already?
Not yet, is the proposed fix, under review now


Question: handler = genericapifilters.WithAuthentication(handler, authenticationInfo.Authenticator, failedHandler, nil) } handler = genericapifilters.WithRequestInfo(handler, requestInfoResolver) <ins> handler = genericapifilters.WithCacheControl(handler) </ins> handler = genericfilters.WithPanicRecovery(handler) return handler

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: func (p *JSONPathPrinter) HandledResources() []string { return []string{} } <ins> // formatEventSource formats EventSource as a comma separated string excluding Host when empty func formatEventSource(es api.EventSource) string { EventSourceString := []string{es.Component} if len(es.Host) > 0 { EventSourceString = append(EventSourceString, es.Host) } return strings.Join(EventSourceString, ", ") } </ins>

Greetings, currently, when doing then in column, the ui result is maybe can do something like this instead? remove then join it with
How about something like ?
Greetings, thanks for the interest, I thought I may throw hacking on it. btw, how about for ?
For we can just print since it's master's component.
Hi looks like there is the same issue reported on the OpenShift Origin github: It would make more sense to solve that upstream. Should we do something together to solve it?
There is a pull request for that:


Question: return false, []algorithm.PredicateFailureReason{ErrNodeUnschedulable}, nil } <ins> return true, nil, nil } // PodToleratesNodeTaints checks if a pod tolerations can tolerate the node taints func PodToleratesNodeTaints(pod *v1.Pod, meta algorithm.PredicateMetadata, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { if nodeInfo == nil || nodeInfo.Node() == nil { return false, []algorithm.PredicateFailureReason{ErrNodeUnknownCondition}, nil } </ins> return podToleratesNodeTaints(pod, nodeInfo, func(t *v1.Taint) bool { // PodToleratesNodeTaints is only interested in NoSchedule and NoExecute taints. return t.Effect == v1.TaintEffectNoSchedule || t.Effect == v1.TaintEffectNoExecute

DaemonSet should not respect node's field. It's because cluster operators often want system components to be running and ready before workloads run on a given node. OTOH, DaemonSet does respect NoSchedule taints. This behavior is broken by , because DaemonSet share some predicates code with default scheduler.
/assign
Added this to 1.10 milestone, because it's a regression introduced in 1.10.
can you (or a SIG lead) add status/approved-for-milestone please? I'll add the priority. /priority important-soon
how is this use case satisfied by the daemonset controller creating pods and depending on the scheduler to assign them to nodes? having the scheduler ignore the unschedulable field seems nonsensical
I'd like get this as limitation for alpha feature; when upgrade to beta, I'd like to use taint to replace un-scheduable, so DaemonSet pod can tolerate it on-demand.
Escalating to critical-urgent, since we are now in Code Freeze. Please update daily with status, thanks! /priority critical-urgent /remove-priority important-soon
In short term, scheduler can check of pods and ignore bit of nodes for DS pods. Long term, when TaintNodesByCondition is stable, we should replace bit by taint and have DS controller add toleration for the taint when creating pods.
That doesn't seem proper. Making pods behave differently depending on the type of their creating object is very counterintuitive. That seems more reasonable. I'm less certain about auto- tolerations
- indeed there's some work currently to move from to a taint based behaviour, tracked by


Question: <ins> //go:build !linux // +build !linux /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "runtime" "k8s.io/klog/v2" ) // chown changes the numeric uid and gid of the named file. // This is a no-op on unsupported platforms. func (w *AtomicWriter) chown(name string, uid, _ /* gid */ int) error { klog.Warningf("%s: skipping change of Linux owner %v for file %s; unsupported on %s", w.logContext, uid, name, runtime.GOOS) return nil } </ins>

Install a cluster with v1.29.4 with linux node as master and worker node as windows kubeadm upgrade apply v1.30.0 fails with --ignore-preflight-errors` Describe Pod events: It seems that previously the pod was Pending as well, but this was ignored, because the jobs was successfully deleted in defer and return value was overridden with nil. Upgrade health check job expected to be scheduled only on Linux nodes using node selectors, rather than on Windows nodes. See What happened? No response_ <detailskubeadm version (use kubeadm version): v1.30.0 Environment: Kubernetes version (use kubectl version): v1.30.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 22.04.2 LTS, Windows Server 2022 Datacenter Kernel (e.g. uname -a): 5.19.0-1025-aws Container networking plugin (CNI): calico </details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in 1.30.1 /close
Closing this issue. <detailsIn response to : kubeadm version (use kubeadm version): v1.30.0 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubeadm
we are seeing this issue for 1.30.1 version too with upgrade health check pods getting scheduled on windows worker nodes.
looks like I was mistaken and this issue is different. but the health check uses the pause image, which is multi os, supposedly. the error is strange. can you show the full log output. we might have to apply a node selector on the job spec and backport to 1.31 another bug fix.

I'm not sure if we should add a node selector on job spec. the job uses the pause image to create a container. It doesn't have specific requirements for the node. After I look at the projected volume plugin, I'm confused why the above event occurs. On windows nodes, the function is empty. It doesn't do anything. On linux nodes, the function will set the owner of the volume. FYI: If a log file from the kubelet is provided, it will be helpful to understand the issue.
thanks for checking I also think there should not be an error. maybe we are hitting a core kubelet bug. is the kubelet on the windows node also 1.30?
please ignore the above comment, it's incorrect. I may find the root cause. It's a bug in kubelet when kubelet is running on Windows and the pod configures the SecurityContext with a RunAsUser.
/sig storage /sig windows
/assign I'm trying to fix the kubelet bug.
so this is not a kubeadm bug and we can move it back to the k/k repo?
/transfer kubernetes
/retitle kubelet on Windows fails if a pod has SecurityContext with RunAsUser.
Because of historical reasons, the and in is only designed for the Linux, not Windows. See for more details. adds a new field to the PodSpec, which can be used to identify the OS of the pod. It's an optional field and no default value. If unset, the above fields are not restricted by the api-server. In other words, the following job is allowed to be created. But it will fails to start on Windows nodes. And the root cause is that the is called by the . To fix this issue, the AtomicWriter can check the current OS of the node and skip the if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. I'm not sure if the failing job is expected or not. If it's expected, kubeadm has to change the job manifest to make it work on only Linux nodes. Why I ask this question is that I saw , which is used to set the ownership of the volume, if the os is Windows, it will become a no-op.

/cc
/cc
/triage accepted
slack discussion is happening here: but we have agreed that this is a kubelet problem. on the kubeadm side you can use the --ignore-preflight-errors=CreateJob option to skip the check. unclear when or if a kubelet fix will be and to what versions it will be available.
To fix this issue, the AtomicWriter can check the current OS of the node and skip the os.Chown if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. please see: do you want to upstream your carry?
thank you. I raise a PR and add as a co-author. I add an e2e test. do you know which job can test windows?
is the e2e test job for Windows


Question: } By("updating the annotation value") <del> var podClient *framework.PodClient podClient = f.PodClient() </del> podClient.Update(podName, func(pod *v1.Pod) { pod.ObjectMeta.Annotations["mysubpath"] = "mynewpath" })

Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test


Question: return nil } <ins> st, dt := sv.Type(), dv.Type() </ins> switch st.Kind() { case reflect.String: if dt.Kind() == reflect.Interface && dv.NumMethod() == 0 {

Kubernetes version (use ): master. What happened: Unstructured converter from does not roundtrip . What you expected to happen: When I call on a which is backed by it should basically make a copy of the content (can be optimized btw using deep copy ) and not a json object with key. How to reproduce it (as minimally and precisely as possible): See the test in my branch Test output: Anything else we need to know: For some reason changed the argument types from to . I guess the intent is to make the converter a more generic tool. But based on the previous signature I assume I'm doing the right thing. Actually, my code works with and I don't know if the actual object behind the interface is an or not - I just pass it to the converter and it works for usual API types, but not for unstructured. Also it would be useful to have a more type-safe version of the converter (i.e. the previous signature) that just delegates to a lower-level converter that does not care about types.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery
/kind bug
/assign
Looked into this issue just now. It is caused by the pointer vs non-pointer receiver of the method - similar to . In detail: is passed as to when it calls passing as . if passed value implements (has method ) does not implement this interface, does. Because on it . So perhaps the fix is to do the dereferencing inside of the ? Or maybe the same way as - check if the value is addressable, get a pointer to the value and then see if it matches the interface (I dunno if it will work that way)? In any case, the check whether a value implements the interface should be done on both the value itself () and the pointer to the value (; if the value is addressable) to cover both valid cases. This is what stdlib package does. Right now it is broken - it dereferences the pointer and then checks if the thing implements the interface. Does it make sense?

Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
does this helps? Look like we need to do MarshalJSON on non-pointers for it to work. I didn't get into the details but if that is the case then making Unstructured's MarshalJSON() receiver, a object instead of pointer should fix your problem, right?
Only if we don't fix the actual cause of the bug which I described above. From the stack overflow question that is mentioned in the link you provided: This code is buggy - to work it should be passing into and into . Yes, json package tries to take the address of the passed (so a pointer receiver should satisfy the interface check) but it cannot take an address of a function argument ( will return false). So, pointer receiver will be invoked if either is a field in a struct (so that an address can be taken via reflection) and a struct is passed (either by value or via a pointer) to / OR is passed. Please take a look at - both issues have the same cause.
I'm happy to take a stab at it and submit a PR but I don't want to spend my time if it is not going to get merged - if we could merge first to make sure people who can approve/lgtm are happy with the approach taken. I strongly believe it is the right thing to do - that is how standard library does it and, well, it is just logical and makes sense :)


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> FROM k8s.gcr.io/kube-cross:v1.10.4-1 as build_k8s_1_7_sample_apiserver </del> <ins> FROM k8s.gcr.io/kube-cross:v1.10.4-1 as build_k8s_1_10_sample_apiserver </ins> ENV GOPATH /go RUN mkdir -p ${GOPATH}/src ${GOPATH}/bin

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: func IsValidPortNum(port int) bool { return 0 < port && port < 65536 } <ins> // IsValidIP tests that the argument is a valid IPv4 address. func IsValidIP(value string) bool { return net.ParseIP(value) != nil && net.ParseIP(value).To4() != nil } </ins>

When spinning up VMs w/ kube, I attempted to use but this fails. ironically, 127.0.0.1 succeeds (internally you can then access a kube service via , for example, but you cannot do this if you try to bind to 0.0.0.0. binding works to 127.0.0.1, but is a horrible idea (since this IP wont be reachable by the outside world), so in that case kube should print a clear message on why this is the case and fail. binding to 0.0.0.0 fails right out of the bat, so in this case, also, kube should fail immediately. To be clear, im suggesting that these should fail at the service submission layer. Possibly a module for validating logic of submitted services/pods , which extends syntactical validation is in order if one doesn't already exist?
Please also see
Since this is very similar to I'm gonna take this.


Question: serviceName = "kubelet" ) <del> func initForOS(windowsService bool) error { </del> <ins> // getPriorityValue returns the value associated with a Windows process priorityClass // Ref: https://docs.microsoft.com/en-us/windows/win32/cimwin32prov/setpriority-method-in-class-win32-process func getPriorityValue(priorityClassName string) uint32 { var priorityClassMap = map[string]uint32{ "IDLE_PRIORITY_CLASS":         uint32(64), "BELOW_NORMAL_PRIORITY_CLASS": uint32(16384), "NORMAL_PRIORITY_CLASS":       uint32(32), "ABOVE_NORMAL_PRIORITY_CLASS": uint32(32768), "HIGH_PRIORITY_CLASS":         uint32(128), "REALTIME_PRIORITY_CLASS":     uint32(256), } return priorityClassMap[priorityClassName] } func initForOS(windowsService bool, windowsPriorityClass string) error { priority := getPriorityValue(windowsPriorityClass) if priority == 0 { return fmt.Errorf("unknown priority class %s, valid ones are available at "+ "https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities", windowsPriorityClass) } kubeletProcessHandle := windows.CurrentProcess() // Set the priority of the kubelet process to given priority klog.Infof("Setting the priority of kubelet process to %s", windowsPriorityClass) if err := windows.SetPriorityClass(kubeletProcessHandle, priority); err != nil { return err } </ins> if windowsService { return service.InitService(serviceName) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil } <del> func (lbaas *LbaasV2) createLoadBalancer(service *v1.Service, name string) (*loadbalancers.LoadBalancer, error) { </del> <ins> func (lbaas *LbaasV2) createLoadBalancer(service *v1.Service, name string, internalAnnotation bool) (*loadbalancers.LoadBalancer, error) { </ins> createOpts := loadbalancers.CreateOpts{ Name:        name, Description: fmt.Sprintf("Kubernetes external service %s", name),

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently openstack cloud provider just support for specifying internal LoadBalancerIP. I think that is unreasonable, actually user just use a floating ip to access a LoadBalancer service. So it is necessary to spport for specifying external LoadBalancerIP. What you expected to happen: Support for specifying external LoadBalancerIP on openstack Anything else we need to know?: Currently LoadBalancerIP can be specified in service.Spec.LoadBalancerIP, we can add annotation to distinguish between internal and external.
/sig openstack /assign I'd like to work for it.


Question: return []net.Interface{p2pIntf}, nil } <ins> // Interface with link locals and loopback interface with global addresses type linkLocalLoopbackNetworkInterface struct { } func (_ linkLocalLoopbackNetworkInterface) InterfaceByName(intfName string) (*net.Interface, error) { if intfName == LoopbackInterfaceName { return &loopbackIntf, nil } return &upIntf, nil } func (_ linkLocalLoopbackNetworkInterface) Addrs(intf *net.Interface) ([]net.Addr, error) { var ifat []net.Addr ifat = []net.Addr{addrStruct{val: "169.254.162.166/16"}, addrStruct{val: "fe80::200/10"}} if intf.Name == LoopbackInterfaceName { ifat = []net.Addr{addrStruct{val: "::1/128"}, addrStruct{val: "127.0.0.1/8"}, // global addresses on loopback interface addrStruct{val: "10.1.1.1/32"}, addrStruct{val: "fd00:1:1::1/128"}} } return ifat, nil } func (_ linkLocalLoopbackNetworkInterface) Interfaces() ([]net.Interface, error) { return []net.Interface{upIntf, loopbackIntf}, nil } // Interface and loopback interface with global addresses type globalsNetworkInterface struct { } func (_ globalsNetworkInterface) InterfaceByName(intfName string) (*net.Interface, error) { if intfName == LoopbackInterfaceName { return &loopbackIntf, nil } return &upIntf, nil } func (_ globalsNetworkInterface) Addrs(intf *net.Interface) ([]net.Addr, error) { var ifat []net.Addr ifat = []net.Addr{addrStruct{val: "169.254.162.166/16"}, addrStruct{val: "fe80::200/10"}, addrStruct{val: "192.168.1.1/31"}, addrStruct{val: "fd00::200/127"}} if intf.Name == LoopbackInterfaceName { ifat = []net.Addr{addrStruct{val: "::1/128"}, addrStruct{val: "127.0.0.1/8"}, // global addresses on loopback interface addrStruct{val: "10.1.1.1/32"}, addrStruct{val: "fd00:1:1::1/128"}} } return ifat, nil } func (_ globalsNetworkInterface) Interfaces() ([]net.Interface, error) { return []net.Interface{upIntf, loopbackIntf}, nil } </ins> // Unable to get IP addresses for interface type networkInterfaceFailGetAddrs struct { }

What happened: When attempting to kubeadm join a control plane node or worker on a machine with global IPv4 unicast addresses on a loopback address (router to the host w/ BGP Unnumbered), the join fails with the following output: See this issue for details : Also: What you expected to happen: The correct IPv4 address should be selected from the loopback adapter, or we should be able to specify an IPv4 address to use for egress. The IPv6 addresses on the network adapters themselves are only used to establish upstream peering. IPv4 nexthops are fake 169.254.x.x addresses artificially programmed via RFC5549 into the ARP table. How to reproduce it (as minimally and precisely as possible): On a node with the following conditions: Routing to the host with FRR IPv6 link locals on physical network interfaces No IPv4 addresses on physical network interfaces Unicast IPv4 on the loopback adapter IPv4 default route present via RFC5549 Attempt to kubeadm join a node, and it will fail. Anything else we need to know?: We've been patching kubeadm with , which works as expected. There was a fix applied to allow initing control plane nodes w/ bind address 127.0.0.1 which would then select an IPv4 from the loopback, but it doesn't appear to have fixed the joining issues. Pinging and by request on kubeadm issues. Environment: Kubernetes version (use ): 1.17.12 Cloud provider or hardware configuration: baremetal OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1127.19.1.el7.x86_64 Install tools: kubeadm Network plugin and version (if this is a network-related bug): Calico Others: FRR for peering with routers on physical network interfaces
/sig api-machinery
is defined but we should note the same function is used for the --bind-address of the kube-apiserver. /sig network
/assign
/triage accepted


Question: container := api.Container{ Ports: []api.ContainerPort{ { <del> Name:     "foo", HostPort: 8080, </del> <ins> Name:          "foo", ContainerPort: 8080, </ins> }, { <del> Name:     "bar", HostPort: 9000, </del> <ins> Name:          "bar", ContainerPort: 9000, </ins> }, }, }

When using probe, define the port by number, it works. But if the port is defined by name like below, it doesn't work. it seems should return other than .


Question: "x=a,y=b,z=c", "", "x!=a,y=b", <ins> "x=", "x= ", "x=,z= ", "x= ,z= ", </ins> } testBadStrings := []string{ "x=a||y=b",

When polling a k8s master with the response is an HTTP 500. But even if this selector is invalid, it should return a 4xx code instead.
Do you get 500 without the label selector? cc
No, and not with a value like "environment%3Dfoobar". But if I remove "foobar" or replace it with an url-encoded space, i.e. "environment%3D%20" or "environment%3D+" I get a 500. I suspect that the value (after the colon) is stripped of whitespace, and if the result is the empty string, this error happens.
Bumping priority to P1 since this is definitely a bug and we should fix it.
you had five days, I'm kicking you off of this & giving to Chao. :)


Question: } } <ins> func TestStatus(t *testing.T) { _, s, closeFn := framework.RunAMaster(nil) defer closeFn() u := s.URL + "/apis/batch/v1/namespaces/default/jobs/foo" resp, err := http.Get(u) if err != nil { t.Fatalf("unexpected error getting %s: %v", u, err) } if resp.StatusCode != http.StatusNotFound { t.Fatalf("got status %v instead of 404", resp.StatusCode) } defer resp.Body.Close() data, _ := ioutil.ReadAll(resp.Body) decodedData := map[string]interface{}{} if err := json.Unmarshal(data, &decodedData); err != nil { t.Logf("body: %s", string(data)) t.Fatalf("got error decoding data: %v", err) } t.Logf("body: %s", string(data)) if got, expected := decodedData["apiVersion"], "v1"; got != expected { t.Errorf("unexpected apiVersion %q, expected %q", got, expected) } if got, expected := decodedData["kind"], "Status"; got != expected { t.Errorf("unexpected kind %q, expected %q", got, expected) } if got, expected := decodedData["status"], "Failure"; got != expected { t.Errorf("unexpected status %q, expected %q", got, expected) } if got, expected := decodedData["code"], float64(404); got != expected { t.Errorf("unexpected code %v, expected %v", got, expected) } } </ins> func TestWatchSucceedsWithoutArgs(t *testing.T) { _, s, closeFn := framework.RunAMaster(nil) defer closeFn()

We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


Question: return p.parseInsideAction(cur) } <del> // parseRecursive scans the recursive desent operator .. </del> <ins> // parseRecursive scans the recursive descent operator .. </ins> func (p *Parser) parseRecursive(cur *ListNode) error { <ins> if lastIndex := len(cur.Nodes) - 1; lastIndex >= 0 && cur.Nodes[lastIndex].Type() == NodeRecursive { return fmt.Errorf("invalid multiple recursive descent") } </ins> p.pos += len("..") p.consumeText() cur.append(newRecursive())

What happened: This issue was originally reported via the Kubernetes HackerOne bug bounty by HackerOne user but we have decided that due to the high privileges required, it can be addressed publicly. A user able to create CRDs could create a malicious CRD such that listing CRs will cause enormous amounts of CPU usage on the API server. What you expected to happen: The function should not expand strings with more than two consecutive periods. Either it should collapse additional repeated periods or it should return an error. How to reproduce it (as minimally and precisely as possible): /kind bug /area apiserver /area security /committee product-security /priority important-longterm
How about we take it a step further by rejecting the jsonpath which contains when parsing it? as more than 2 periods seem to make no sense IMO
good idea. I have posted to do what you suggest.
/sig api-machinery
/cc


Question: errorStatus  = "ERROR" ServiceAnnotationLoadBalancerFloatingNetworkId = "loadbalancer.openstack.org/floating-network-id" <ins> // ServiceAnnotationLoadBalancerInternal is the annotation used on the service // to indicate that we want an internal loadbalancer service. // If the value of ServiceAnnotationLoadBalancerInternal is false, it indicates that we want an external loadbalancer service. Default to true. ServiceAnnotationLoadBalancerInternal = "service.beta.kubernetes.io/openstack-internal-load-balancer" </ins> ) // LoadBalancer implementation for LBaaS v1

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently openstack cloud provider just support for specifying internal LoadBalancerIP. I think that is unreasonable, actually user just use a floating ip to access a LoadBalancer service. So it is necessary to spport for specifying external LoadBalancerIP. What you expected to happen: Support for specifying external LoadBalancerIP on openstack Anything else we need to know?: Currently LoadBalancerIP can be specified in service.Spec.LoadBalancerIP, we can add annotation to distinguish between internal and external.
/sig openstack /assign I'd like to work for it.


Question: BeforeEach(func() { framework.SkipUnlessNodeCountIsAtLeast(minNodes) <ins> cs = f.ClientSet ns = f.Namespace.Name </ins> <del> podClient = f.ClientSet.Core().Pods(f.Namespace.Name) nodeClient = f.ClientSet.Core().Nodes() nodes = framework.GetReadySchedulableNodesOrDie(f.ClientSet) </del> <ins> podClient = cs.Core().Pods(ns) nodeClient = cs.Core().Nodes() nodes = framework.GetReadySchedulableNodesOrDie(cs) </ins> Expect(len(nodes.Items)).To(BeNumerically(">=", minNodes), fmt.Sprintf("Requires at least %d nodes", minNodes)) host0Name = types.NodeName(nodes.Items[0].ObjectMeta.Name) host1Name = types.NodeName(nodes.Items[1].ObjectMeta.Name)

When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .


Question: stop := make(chan struct{}) namespaceController.Run(stop) <del> ns1 := api_v1.Namespace{ ObjectMeta: api_v1.ObjectMeta{ Name:     "test-namespace", SelfLink: "/api/v1/namespaces/test-namespace", }, } </del> // Test add federated namespace. namespaceWatch.Add(&ns1) createdNamespace := GetNamespaceFromChan(cluster1CreateChan)

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: assert.Equal(t, len(result), 0, "TestCase[%d]: %s", i, c.desc) } } <ins> func TestServiceOwnsPublicIP(t *testing.T) { tests := []struct { desc        string pip         *network.PublicIPAddress clusterName string serviceName string expected    bool }{ { desc:        "false should be returned when pip is nil", clusterName: "kubernetes", serviceName: "nginx", expected:    false, }, { desc: "false should be returned when service name tag doesn't match", pip: &network.PublicIPAddress{ Tags: map[string]*string{ serviceTagKey: to.StringPtr("nginx"), }, }, serviceName: "web", expected:    false, }, { desc: "true should be returned when service name tag matches and cluster name tag is not set", pip: &network.PublicIPAddress{ Tags: map[string]*string{ serviceTagKey: to.StringPtr("nginx"), }, }, clusterName: "kubernetes", serviceName: "nginx", expected:    true, }, { desc: "false should be returned when cluster name doesn't match", pip: &network.PublicIPAddress{ Tags: map[string]*string{ serviceTagKey:  to.StringPtr("nginx"), clusterNameKey: to.StringPtr("kubernetes"), }, }, clusterName: "k8s", serviceName: "nginx", expected:    false, }, { desc: "false should be returned when cluster name matches while service name doesn't match", pip: &network.PublicIPAddress{ Tags: map[string]*string{ serviceTagKey:  to.StringPtr("web"), clusterNameKey: to.StringPtr("kubernetes"), }, }, clusterName: "kubernetes", serviceName: "nginx", expected:    false, }, { desc: "true should be returned when both service name tag and cluster name match", pip: &network.PublicIPAddress{ Tags: map[string]*string{ serviceTagKey:  to.StringPtr("nginx"), clusterNameKey: to.StringPtr("kubernetes"), }, }, clusterName: "kubernetes", serviceName: "nginx", expected:    true, }, } for i, c := range tests { owns := serviceOwnsPublicIP(c.pip, c.clusterName, c.serviceName) assert.Equal(t, owns, c.expected, "TestCase[%d]: %s", i, c.desc) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We deployed two Kubernetes clusters to the same Azure Resource Group. Created services of load balancer type on both clusters. One of the services never received a public IP. Upon investigation, we noticed that the first Kubernetes cluster deleted the Azure Public IP that was created for the service by the other cluster. Apparently, all public IPs in the resource group are retrieved: then public IPs which don't match this criteria: are deleted. This logic besides the name of the kubernetes service should also consider the K8s cluster where the public IP originated from to avoid stepping on each other's toes when multiple clusters share the same resource group. What you expected to happen: In case multiple clusters run in the same resource group a cluster should delete only resources created by itself. How to reproduce it (as minimally and precisely as possible): Create two Kubernetes clusters (1.14) with different cluster names to the same resource group than create kubernetes services of load balancer type in each cluster. Anything else we need to know?: Environment: Kubernetes version (use ): 1.4 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks for reporting the issue, this is the bug we should fix for sharing resource group. /sig azure /kind bug /assign


Question: selector: matchLabels: k8s-app: elasticsearch-logging <del> version: v7.1.1 </del> <ins> version: v7.2.0 </ins> template: metadata: labels: k8s-app: elasticsearch-logging <del> version: v7.1.1 </del> <ins> version: v7.2.0 </ins> spec: serviceAccountName: elasticsearch-logging containers: <del> - image: quay.io/fluentd_elasticsearch/elasticsearch:v7.1.1 </del> <ins> - image: quay.io/fluentd_elasticsearch/elasticsearch:v7.2.0 </ins> name: elasticsearch-logging <ins> imagePullPolicy: Always </ins> resources: # need more cpu upon initialization, therefore burstable class limits:

What happened: On latest fluentd-elasticsearch, the elasticsearch replicas seems failed to form a valid cluster with the service discovery helper written in golang. What you expected to happen: Successfully boot up an elasticsearch cluster I guess. How to reproduce it (as minimally and precisely as possible): Just use the yamls in and the elasticsearch-logging pods will keep spamming warning messages. I think thats because options are deprecated in elasticsearch 7.x.x Anything else we need to know?: IMHO I think thats all. Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig instrumentation
Thanks


Question: // Note that containerID is only needed when following the log, or else // just pass in empty string "". func ReadLogs(ctx context.Context, path, containerID string, opts *LogOptions, runtimeService internalapi.RuntimeService, stdout, stderr io.Writer) error { <ins> // fsnotify has different behavior for symlinks in different platform, // for example it follows symlink on Linux, but not on Windows, // so we explicitly resolve symlinks before reading the logs. // There shouldn't be security issue because the container log // path is owned by kubelet and the container runtime. evaluated, err := filepath.EvalSymlinks(path) if err != nil { return fmt.Errorf("failed to try resolving symlinks in path %q: %v", path, err) } path = evaluated </ins> f, err := os.Open(path) if err != nil { return fmt.Errorf("failed to open log file %q: %v", path, err)

doesn't work for Windows node. After some investigation, I think the problem is that for Docker the log file reads is a symlink to the actual docker container log file. On linux, I believe fsnotify follows symlink by default , so it works on linux. However, on windows I believe it doesn't follow the symlink. I tried to replace the symlink file with an actual file and append log to it, will start working after that. The proposed solution is to explicitly follow the symlink in Kubelet, so that we get more consist behavior across different platforms.
/cc
Note that this happened to be fixed by in HEAD. But we may still want to follow the symlink which is a more proper fix.


Question: }) } else if protocol != v1.ProtocolUDP && protocol != v1.ProtocolSCTP { // we only add the expected probe if we're doing TCP <ins> if probeProtocol == "" { probeProtocol = string(*probeProto) } var actualPath *string if !strings.EqualFold(probeProtocol, string(network.ProbeProtocolTCP)) { if requestPath != "" { actualPath = to.StringPtr(requestPath) } else { actualPath = to.StringPtr("/healthz") } } </ins> expectedProbes = append(expectedProbes, network.Probe{ Name: &lbRuleName, ProbePropertiesFormat: &network.ProbePropertiesFormat{ <del> Protocol:          *probeProto, </del> <ins> Protocol:          network.ProbeProtocol(probeProtocol), RequestPath:       actualPath, </ins> Port:              to.Int32Ptr(port.NodePort), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2),

What would you like to be : Support for additional load balancer health probe configurations (specifically HTTP and HTTPS) for Azure Load Balancers. Why is this needed: The generated Azure Load Balancers for type services only support TCP health probes. Those guarantee that the backend pool consists of nodes listening on the target port but do not validate whether or not anything is responding. We've also noticed elevated volumes SSL handshake errors generated by the TCP health checks targeting 443 when using more verbose log level configurations in ingress-nginx. It would be valuable to configure the type of health probe used by the AKS-created Azure Load Balancer.
/sig azure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any update on this?
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about this feature requests?
I had a look at a possible implementation. I did not create a PR yet, because the relevant files are in and the said there should only be bug fixes, no new features. In , there is some logic which would lead to the load balancer created with protocol HTTP (which is what we want) The check is from here: and the "RequestsOnlyLocalTraffic" is about the : However, I do not understand the relationship here, why setting should result in HTTP health check. What I think we could do is add a new annotation like and when this is set to , the health check will use the HTTP protocol in .

We could also add an additional annotation to set the property of the health check probe, this could be used so that people can set the path to or similar.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We have a similar requirement for having an http health probe with a custom path. According to comment, we could get an http probe by setting but our service expects health probes on a different path, not . Looks like proposed solution would work for us too. Are there any workarounds to achieve this currently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/pkg/controller/replicaset"

[X] test approval issue : [x] test pr : [x] two weeks soak end date : 2 June 2021 [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints which are untested. Create a ReplicaSet List all Replicasets DeleteCollection of ReplicaSets Varify the ReplicaSets have been deleted Using an existing conformance test as a template for a which validates that two new endpoints are hit. example endpoint | useragent --------------------------------------------|
createAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets deleteAppsV1CollectionNamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets listAppsV1ReplicaSetForAllNamespaces | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets readAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets (4 rows) If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
/triage accepted For Release 1.22
/assign
Dear When I run It only returns not listAppsV1ReplicaSetForAllNamespaces. What does this mean? What should I do? Regarding What should I see after implementing the tests? All 4 rows? Right now I see 0 rows when running it (without editing code) Lastly, could you tell me which parts need to be addressed in the ? Right now, it looks like the template does exactly what you want to write for this test.
This issue is created to show a planned PR to the conformance meeting. It is just a proof of concept before creating the conformance PR. What we do: I create a local cluster and create a test. In the cluster, I have running to see what user agents are hit by the test. Before I run the test I delete all cluster logs.

Then I run the test. This is my test: Once the test pass I run the APISnoop query to see which user-agents are hit. I in this case 4 API's are hit, but I am only interested in and as they are the ones yet.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: AccessModes: []v1.PersistentVolumeAccessMode{ v1.ReadWriteOnce, v1.ReadOnlyMany, <del> v1.ReadWriteMany, </del> }, Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-ubuntu-gce-containerd Which test(s) are flaking: [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a persistent volume claim with a storage class. [sig-storage] Testgrid link: Reason for failure: Anything else we need to know: links to appreciated Few other instances:
/sig api-machinery
Thank you for reporting this. Worth noting I think, that the test has failed with timeout across a multiple jobs (the triage link you provided lists them there) Also, there are additional failure reasons present on the triage page when run from e2e-kops-* jobs Again more details at the bottom of the triage page for . and /cc
/cc
looks like this test still periodically times out, would be great if someone could take a look
can you look and find an owner?
Ensuring resource quota status captures persistent volume claim creation Nov 8 00:00:58.461: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:00.449: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:03.123: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:04.545: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:06.529: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:08.499: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:10.471: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:12.624: INFO: resource requests.storage, expected 1Gi, actual 0 Nov 8 00:01:14.612: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:16.494: INFO: resource persistentvolumeclaims, expected 1,

actual 0 Nov 8 00:01:18.661: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:20.488: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:22.432: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:24.963: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:26.634: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:26.959: INFO: resource persistentvolumeclaims, expected 1, actual 0 Nov 8 00:01:26.959: FAIL: Unexpected error: <errors.errorString | 0xc0001fa1f0: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace () +0x8e5 (0xc002c66300) +0x345 (0xc002c66300) +0x2b testing.tRunner(0xc002c66300, 0x4dca5d0) +0xef created by testing.(T).Run +0x386 [AfterEach] [sig-api-machinery] ResourceQuota 1mSTEP0m: Collecting events from namespace "resourcequota-383". 1mSTEP0m: Found 1 events. Nov 8 00:01:27.139: INFO: At 2020-11-08 00:00:57 +0000 UTC - event for test-claim: {persistentvolume-controller } ProvisioningFailed: Failed to provision volume with StorageClass "standard": invalid AccessModes [ReadWriteOnce ReadOnlyMany ReadWriteMany]: only AccessModes [ReadWriteOnce ReadOnlyMany] are supported
As the errors pointed out, the PVC failed to be provisioned due to invalid access mode specified.


Question: // Start syncing or monitoring node status. if syncNodeStatus { go util.Forever(func() { <del> if err = nc.SyncProbedNodeStatus(); err != nil { </del> <ins> if err := nc.SyncProbedNodeStatus(); err != nil { </ins> glog.Errorf("Error syncing status: %v", err) } }, period) } else { go util.Forever(func() { <del> if err = nc.MonitorNodeStatus(); err != nil { </del> <ins> if err := nc.MonitorNodeStatus(); err != nil { </ins> glog.Errorf("Error monitoring node status: %v", err) } }, nodeMonitorPeriod)

stack trace here: it's due to a shared variable:
Hi thanks for reporting the issue with a patch! Do you want to submit a PR so that you can take the credit for your fix? :)
done , Yu-Ju Hong wrote: 585.241.9488 (voice) 650.649.6071 (fax)
FYI, I think the same bug may still exist in your new PR.
There are multiple p
There are multiple PRs in flight, so I don't know which one you have in mind. One of them removes probing logic from controller altogether.
I was referring to , but I didn't read that PR closely so the bug may longer exist there. Again, just a FYI. :)


Question: } for i, pType := range spec.PolicyTypes { policyPath := fldPath.Child("policyTypes").Index(i) <del> for _, p := range spec.PolicyTypes { if !allowed.Has(string(p)) { allErrs = append(allErrs, field.NotSupported(policyPath, pType, []string{string(networking.PolicyTypeIngress), string(networking.PolicyTypeEgress)})) } </del> <ins> if !allowed.Has(string(pType)) { allErrs = append(allErrs, field.NotSupported(policyPath, pType, []string{string(networking.PolicyTypeIngress), string(networking.PolicyTypeEgress)})) </ins> } } return allErrs

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Create a NetworkPolicy with one typo PolicyType and a correct one. The apiserver will report error but will also say the correct one is unsupported. Output: It says is unsupported. What you expected to happen: It shouldn't say the correct policyType is unsupported. How to reproduce it (as minimally and precisely as possible): Apply the above yaml. Anything else we need to know?: It's because an excess nested loop is to validation function: Environment: Kubernetes version (use ): v1.16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved


Question: return err } <del> if err := watchHandler(start, w, r.store, r.expectedType, r.expectedGVK, r.name, r.expectedTypeName, r.setLastSyncResourceVersion, r.clock, resyncerrc, stopCh); err != nil { </del> <ins> err = watchHandler(start, w, r.store, r.expectedType, r.expectedGVK, r.name, r.expectedTypeName, r.setLastSyncResourceVersion, r.clock, resyncerrc, stopCh) retry.After(err) if err != nil { </ins> if err != errorStopRequested { switch { case isExpiredError(err):

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.


Question: It("Pod should avoid to schedule to node that have avoidPod annotation", func() { nodeName := nodeList.Items[0].Name // make the nodes have balanced cpu,mem usage <del> createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5) </del> <ins> err := createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5) framework.ExpectNoError(err) </ins> By("Create a RC, with 0 replicas") rc := createRC(ns, "scheduler-priority-avoid-pod", int32(0), map[string]string{"name": "scheduler-priority-avoid-pod"}, f, podRequestedResource) // Cleanup the replication controller when we are done.

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: dirPodMap[path] = internalPodUid podUidMap[internalPodUid] = externalPodUid podDirCountMap[internalPodUid]++ <del> klog.V(4).Infof("Assigning quota ID %d (%d) to %s", id, ibytes, path) </del> <ins> klog.V(4).Infof("Assigning quota ID %d (request limit %d, actual limit %d) to %s", id, ibytes, fsbytes, path) </ins> return nil } removeProjectID(path, id)

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: // failed by this plugin schedulable. func (pl *NodeName) EventsToRegister() []framework.ClusterEvent { return []framework.ClusterEvent{ <del> {Resource: framework.Node, ActionType: framework.Add}, </del> <ins> {Resource: framework.Node, ActionType: framework.Add | framework.Update}, </ins> } }

We have an e2e test that involves cluster autoscaler; the test creates two pods back to back which forces CA to scale up the cluster to create two nodes also back to back one for each pod. The two nodes are not created at the same time, there is some delay between them. The problem is that the second pod never gets scheduled. We see in the logs a single scheduling attempt for the second pod that attempted to schedule it on the first node and failed because it didn't fit the node's remaining cpu cores. We don't see any attempts after that for the pod and the test times out waiting for that pod to schedule. What changed in the scheduler: Pre 1.22, on most update/add/delete events to pods/nodes/pvc/storageClass resources, we used to move all unschedulable pods back to the queue. Also every we did force move all unschedulable pods back to the queue indiscriminately. This behavior is not efficient, wasted scheduling cycles and since we re-queue based on last scheduling attempt timestamp, it missed up badly with the best effort FIFO behavior we intend to have for pod scheduling.Since 1.22 a couple of things changed such that unschedulable pods are not placed back into the queue on those events indiscriminately; on each such event, we iterate over unschedulable pods and we place them back into the queue if they pass two checks: the first one happens only on node update/add events, it is carried by the function ; it basically runs the kubelet admission logic (node resource, affinity, taints/tolerations and host port checks) the second check runs on all events (including the node ones); this one is a little more complicated and fine grained: for each unschedulable pod we remember which plugins last failed to fit the pod; at the same time plugins register which events could make an unschedulable pod schedulable (e.g. nodeResources filter plugin registers pod delete and node add/update events). A pod is put back into the queue only if the event matches any of the ones registered by the plugins that last failed the pod.

In 1.24 we increased the 60s to (our north star to completely remove this) What went wrong: My theory is that we incorrectly determined that the pod was not eligible to be put back into the queue on events related to the new node; and that the 5min timeout was too long to force moving it back to the queue (I am guessing the whole test timed out), while the 60s was basically masking the problem for 1.22 and 1.23. So what checks are incorrectly failing to move the pod? Here is what happened: Before adding the new node, the pod failed the nodeResource plugin, which only registers for New node was , scheduler receives an add event. The pod was not moved back to the queue because it failed the check on NotReady taint. When the new node became ready, the scheduler receives a node update event, but since it is a taint update event, it didn't match the events that the nodeResources plugin registered for. How to fix it: Since we have the ; pods that previously failed because of NodeResources, NodePorts, TaintsAndTolerations and NodeAffinity (the predicates we already ran in ) should always be placed back into the queue; basically those plugins should register for all node update events. /sig scheduling /cc The second pod to schedule. See above. No response 24 GKE <details</details<details</details<details</details<details</details
/triage accepted


Question: expectedList := &unstructured.UnstructuredList{} expectedList.SetGroupVersionKind(listGVK) expectedList.SetResourceVersion("") // by product of the fake setting resource version <ins> expectedList.SetContinue("") </ins> expectedList.Items = []unstructured.Unstructured{u} if diff := cmp.Diff(expectedList, list); diff != "" {

The fake dynamic client does not support mock pagination or the self link. Add ability to propagate the metadata fields from a mock unstructured list to the served list by the fake lister. Repro linked in the MR x-ref. No response <details</detailsN/A N/A <detailsN/A </details<detailsN/A </details<detailsN/A </details
/sig api-machinery
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: requireAtomicSetType:                     requireAtomicSetType(&oldObj.Spec), requireMapListKeysMapSetValidation:       requireMapListKeysMapSetValidation(&oldObj.Spec), preexistingExpressions:                   findPreexistingExpressions(&oldObj.Spec), <ins> versionsWithUnchangedSchemas:             findVersionsWithUnchangedSchemas(obj, oldObj), </ins> celEnvironmentSet:                        environment.MustBaseEnvSet(environment.DefaultCompatibilityVersion()), } return validateCustomResourceDefinitionUpdate(ctx, obj, oldObj, opts)

We were using cel-go v0.12.7 in Kubernetes 1.27 release and cel-go v0.16.0 in Kubernetes 1.28 release. There are many changes from cel-go side between those two releases and some may break backward compatible of CRD validation rules. An example would be the estimated cost for CRD validation rules(Thanks to who brought this to our attention with the following example). In the example CRD validation rules below: It will pass perfect in Kubernetes 1.27 version but fail in Kubernetes 1.28 with error: This is due to the estimator fix happened in cel-go(, ) The changes should not cause backward incompatible of CRD validation rules. OR a stability test suite should be in place to prevent the changes causing backward compatible issue before GA. Apply the above CRD in Kubernetes 1.27 and 1.28, 1.27 cluster should pass and 1.28 cluster should fail. Since the fix in cel-go side makes sense to me and the feature is in Beta now, I am not quite sure how strict we should be on backward compatible. cc As a followup, a stability test suite for estimate cost is worth adding before GA to make sure any future changes be caught before the feature promoted to GA. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I agree that the changes in cel-go should not break the backward compatibility of CRD validation rules, especially for users who have already defined and deployed their custom resources using the previous version of Kubernetes. One suggestion I have is to use a version flag for the CEL library, so that users can specify which version of cel-go they want to use for their CRD validation rules. This way, they can avoid unexpected errors or behavior changes when upgrading their Kubernetes cluster.

For example, they can add something like in their CRD schema to indicate that they want to use the old version of cel-go.
Another suggestion I have is to use a cost limit parameter for the CEL library, so that users can adjust the budget for their CRD validation rules. This way, they can fine-tune the trade-off between performance and complexity of their rules. For example, they can add something like in their CRD schema to indicate that they want to allow more expensive rules.
I agree this is the type of bug fix that is legitimate to make during beta. when the size changes came in, did our existing size estimation test coverage notice this cost increased? If not, is there a systemic change we could make to exercise each type of estimation combination so we have visibility to any future changes. Existing persisted CRD objects should continue to work as-is on upgrade, but the CEL rules and/or schema would be required to fit within fixed cost estimates before making future updates/changes to the CEL rules there should be unit tests ensuring updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates, but can you verify that is the case? That's not workable, both from an implementation perspective and from a fragmentation perspective. We explicitly don't want to allow opting into loopholes / bugs in cost estimation that allow CRD authors to make API servers do excessive amounts of work. Similar to above, the responsibility for API server resource consumption lies with the kube-apiserver admin, not the CRD author, so exposing this in the API is not something I would expect.
/assign
Thank you! I guess the next step would be: the existing test suite regarding with the cel cost evaluation. Add estimated cost tests for every single operation, function, type, etc if it's not existing now. updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates. Currently it is not working like this but would definitely help here. cc , please correct me if I'm wrong :) Those should be done before the feature goes to GA. I am not sure about the policy of a GA feature relying on alpha/beta feature is acceptable or not. is going to Beta in this release. The worst case is we have to wait further until the feature is ready.

I don't think that helps... CRD ratcheting is about validation of the custom resources. This is about validation of the CRD itself.
thanks for clarifying! Are you suggesting while updating CRD, if the rules matching what's in the existing object, we should skip cost estimation check? That makes sense. I could start a PR if my understanding aligns :)
something along those lines, though now that I think about it, other changes in the schema could affect the number of times a CEL expression could be run (dropping or increasing maxItems, maxProperties, etc) we already collect preexisting expressions when updating a CRD in order to allow already-persisted expressions (possibly persisted via a newer API server) making use of new CEL features: I think that is what I was thinking of. I would probably focus on: coverage of the edges of the cost estimation so we don't pick up changes there without knowing coarse-grained skipping of schema/cel validation for a particular version in a CRD for updates which don't change anything under that version's schema (would let someone make unrelated updates to the CRD and leave an existing version's schema / CEL rules alone)
Thank you! We have those work planned for the current cycle. /assign /assign


Question: patchType := reflect.TypeOf(patchV) if originalType != patchType { if !isDeleteList { <ins> if mergeOptions.IgnoreUnmatchedNulls { discardNullValuesFromPatch(patchV) } </ins> original[k] = patchV } continue

Note: I'm using OpenShift. s/oc/kubectl/ below if that's how you roll. I'm using a command similar to the following to idempotently ensure a label is absent on a Namespace object: If the object's contains anything other than , this works fine: the resulting object has no key. However, if is initially absent, this command adds it with an entry with the empty string as a value: Running the command again removes the entry and the . Using this logic in a controller causes a hot loop as we thrash back and forth between the above and no labels. Using a value in a dict in a StrategicMerge patch should idempotently ensure the key is absent, even if the dict is initially absent/empty. I didn't try this on anything other than a Namespace; and I didn't try it on any dict other than . Workaround: Add any other label, as seen above. <details</details<detailsAWS </details<details</details<details</detailsI believe the issue is on the server, but... v0.9.3-0.- </details<details</details
/sig api-machinery
/assign /cc /triage accepted
can you look into it?
It looks that this idempotency problem is fixed in latest releases(after being label as you mentioned). All responses are legit; Kubernetes Version
The bug happens when there are no other labels. Are you saying the "fix" was for kubernetes to always add a label to the object?
That was an incidental change specific to namespaces. I'd expect the issue still exists for other types
you are both right. This is incidental "fix" for namespaces. I'll look at others to find the actual problem.


Question: for group_version in "${GROUP_VERSIONS[@]}"; do generate_version "${group_version}" done <del> "${KUBE_ROOT}/hack/update-swagger-spec.sh" </del>

Ref: verify-swagger-spec is always passing on both travis and jenkins even when the swagger spec is not updated. It fails when run locally if the spec is not updated. Need to figure out why does it not fail on travis and jenkins.
It is failing now: https://travis- All I did was add some echo statements :)
Sent to fix this


Question: ginkgo.By("Analyzing results") // readiness probes are called during pod termination <del> framework.ExpectNoError(results.RunTogether(prefixedName(PreStopPrefix, regular1), prefixedName(ReadinessPrefix, regular1))) </del> <ins> framework.ExpectNoError(results.RunTogetherLhsFirst(prefixedName(PreStopPrefix, regular1), prefixedName(ReadinessPrefix, regular1))) </ins> // liveness probes are not called during pod termination <del> err = results.RunTogether(prefixedName(PreStopPrefix, regular1), prefixedName(LivenessPrefix, regular1)) </del> <ins> err = results.RunTogetherLhsFirst(prefixedName(PreStopPrefix, regular1), prefixedName(LivenessPrefix, regular1)) </ins> gomega.Expect(err).To(gomega.HaveOccurred()) })

master-blocking: ci-node-e2e Recent failures: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is also similar. The fix for that was bumping the delay time.
Hi Vyom - would this be a good first issue because there is a previous similar solution?
I hit the same failure in my . This seems caused because expects the regular container to start before the postStart: I posted a PR() to add to the postStart. However, there are other approaches such as: Introduce as mentioned . Remove from this test case because this check does not look the primary purpose of this testcase.
The test mentioned by is similar to this flake and doesn't use . I can just remove it. I'm also going to add a comment on RunTogether to indicate it expects ordering to be enforced. That being said, shouldn't PostStart be enforced to run after the container is started? /assign
Hey PRs are more than welcome :)
I've submitted the PR, let me know if I didn't follow any best practices or anything else is required, thanks!
Skipped after was merged. Last CI run passed. /close
Closing this issue. <detailsIn response to : Skipped after was merged. Last CI run passed. </details
/reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry.
Reopened this issue. <detailsIn response to : /reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details