Question: // SelectorFromValidatedSet returns a Selector which will match exactly the given Set. // A nil and empty Sets are considered equivalent to Everything(). // It assumes that Set is already validated and doesn't do any validation. <ins> // Note: this method copies the Set; if the Set is immutable, consider wrapping it with ValidatedSetSelector // instead, which does not copy. </ins> func SelectorFromValidatedSet(ls Set) Selector { if ls == nil || len(ls) == 0 { return internalSelector{}

A common pattern in Kubernetes (both internally and in third party usage) is something like: (this is from the ). This is commonly called at a fairly high rate; in our case we would actually end up seeing this account for significant portions of CPU/memory to compute the and run on it. In our controller, as an optimization, we replaced usage of and just used our own simpler label matching code, leading to a 70% speed up in our function (which does more than just match) and 96% allocation size improvement; see I believe similar could be done in Kubernetes directly by having a specialized Selector implementation for simple labels. This would transparently given substantial performance benefits to anyone using the library
/sig api-machinery
is the change here a new kind of selector matching or a tweak to something existing? It's hard to tell if this is proposing a new design to partition types of selectors that needs a kep versus a perf tweak to existing code that is straight-forward.
The proposal is an internal performance improvement to make a specialized implementation of the selector matching internally. So two concrete implementations for 1 shared interface. is a PoC of this. I think it can be extended (by me, or others if someone wants to) to complete it
We discussed this one during our bug triage. There seemed to be the feeling that this is large enough to need a KEP? Would you be up for that? Additionally you could bring it up in our SIG meeting to socialize / discuss it more if you wanted.
The patch shared in helps to explain, it's a lot smaller than the description made it sound. This is a special case for equality-only selectors. Probably still worth bringing up at a sig-meeting, but if if the patch is a good representation, it is more bounded than expected from the issue description.
/triage accepted


Question: return keyring.Lookup(image) } <ins> type FakeKeyring struct { auth docker.AuthConfiguration ok   bool } func (f *FakeKeyring) Lookup(image string) (docker.AuthConfiguration, bool) { return f.auth, f.ok } </ins>

I launched a pod forgetting that I had not uploaded my into the node's folder. Of course the image pull failed but the error message was not helpful: e.g. It should finish off the message with "because credentials for are not available to Kubernetes. This may be because blah"


Question: if IsJobFinished(&job) { return nil } <ins> var manageJobErr error </ins> if pastActiveDeadline(&job) { // TODO: below code should be replaced with pod termination resulting in // pod failures, rather than killing pods. Unfortunately none such solution

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.

First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1.4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs . However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway.

Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me?

if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).


Question: // and may be the result of another HTTP actor. // It supports wrapped errors and returns false when the error is nil. func IsUnexpectedServerError(err error) bool { <del> if status := APIStatus(nil); errors.As(err, &status) && status.Status().Details != nil { </del> <ins> status, ok := err.(APIStatus) if (ok || errors.As(err, &status)) && status.Status().Details != nil { </ins> for _, cause := range status.Status().Details.Causes { if cause.Type == metav1.CauseTypeUnexpectedServerResponse { return true

I have a controller that creates objects. It uses to notice when create fails due to a deleting namespace. This check was working well until I moved it away from each call. In some cases, the error is wrapped using causing the function to always return false. I expected to work on both wrapped and unwrapped errors, like the functions do. This short program shows the issue: I am able to work around it by first calling . This type assertion should be replaced with a call to : /sig api-machinery v1.22.5, but I also see the affected code in master. <details</detailsn/a n/a k3s n/a n/a
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: return filepath.Join(fstore.path, key) } <ins> // ensurePath creates input directory if it does not exist func ensurePath(path string) error { if _, err := os.Stat(path); err != nil { // MkdirAll returns nil if directory already exists return os.MkdirAll(path, 0755) } return nil } </ins> func validateKey(key string) error { if len(key) <= keyMaxLength && keyRegex.MatchString(key) { return nil

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Using to start a cluster with only one node and found that there are some error logs in kubelet as following: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc
/cc
Seems caused by my env due to I cannot access gcr due to the proxy failed, will continue to check...
We should create the checkpoint directory when creating the checkpoint store, or else the list failed error will be quite spammy before a container is created.
Will push a fix soon.


Question: spec: containers: - name: kubedns <del> image: gcr.io/google_containers/kubedns-amd64:1.6 </del> <ins> image: gcr.io/google_containers/kubedns-amd64:1.7 </ins> resources: # TODO: Set memory limits when we've profiled the container for large # clusters, then set request = limit to keep this container in

Now that merged, we need to build a new kube-dns release.
I just put in a PR yesterday to fix the missing changelog entry for kube-dns:1.6


Question: "time" "github.com/stretchr/testify/assert" corev1 "k8s.io/api/core/v1" discovery "k8s.io/api/discovery/v1" apiequality "k8s.io/apimachinery/pkg/api/equality"

My cluster contains about 4w endpointslices. There are 3 related metrics (endpointslicecontrollerendpointsaddedpersynccount, endpointslicecontrollerendpointsremovedpersynccount, endpointslicecontrollerchanges) to indicates endpoint add remove and update. I found that the apiserver periodically receives lots of endpointslice update requests. In terms of the number of requests, the endpointslice has been fully updated. But from the endpointslice resource itself, it seems that it has not been updated, because "" annotation didn't change It seems that there is no need to update the endpointslice with no really changes. On the other hand, a large number of ep slice update requests will stress the apiserver. In order to make the phenomenon more obvious, the min-resync-period parameter of controller-manager can be adjusted to 1 minute. Create one or more empty endpointslices, and then you can find in the apiserver logs that requests to update these endpointslice every one to two minutes. For clusters with lots of empty endpointslices, this bug may affect the stability of it. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
there were different improvements in this area, but since 1.20 is going EOL in 8 days it will be good to know if this happens in 1.21 /cc
I'm glad to recieve your comment. I'm not sure because I only have cluster with version 1.20.10. Expect someone who could provide more detail about it. It is actually happening in my cluster, but I don't know how it happens.
I've seen this in 1.20 clusters as well, but have not seen it in 1.21.

My theory was that this is mostly or entirely fixed by Prior to that, any change to EndpointSlice labels would not trigger an EndpointSlice controller sync loop, but the resync period would result in those EndpointSlices being updated at that point. more recent fix () seems like it could also be relevant, but I don't think it matches what's described here. In that case, I believe we're unnecessarily updating EndpointSlices if something irrelevant in the Pod changes (resource version), but I don't believe there's anything special there that would mean we weren't updating EndpointSlices until the resync period. So with all of that said, I believe this is fixed in 1.21+, but let me know if anyone's able to recreate this in those versions and/or if is actually more relevant than it appears on the surface.
Thanks for your reply I will check it.
I confirmed that the periodic updates in my cluster are all empty endpointslices. There are l lot of IPv6 endpointslice in my cluster. But it shouldn't be updated when endpointslice-controller couldn't list pods by service label selector.
/assign
I found that when endpointslice's endpoint array is empty, endpointslice-controller updates it every periodic resync, although it didn't change. I consider it a bug and would like to fork and fix it. The code below could fix the issue of updating empty endpointslices, but I'm not sure if it will introduce new issues. for i := 0; i < len(slicesToDelete); { if len(slicesToCreate) == 0 { break } sliceToDelete := slicesToDelete[i] slice := slicesToCreate[len(slicesToCreate)-1] // Only update EndpointSlices that are owned by this Service and have // the same AddressType. We need to avoid updating EndpointSlices that // are being garbage collected for an old Service with the same name. // The AddressType field is immutable. Since Services also consider // IPFamily immutable, the only case where this should matter will be // the migration from IP to IPv4 and IPv6 AddressTypes, where there's a // chance EndpointSlices with an IP AddressType would otherwise be // updated to IPv4 or IPv6 without this check.

if sliceToDelete.AddressType == slice.AddressType && ownedBy(sliceToDelete, service) { slicesToCreate = slicesToCreate[:len(slicesToCreate)-1] // We must consider the situation that an empty unchanged endpointslice no need to update. If the one to be deleted and the one to be created are the same endpointslice which is empty, skip the update step. // eg: If there are many headless svc with single stack ipv4 for example, a lot of ipv6 empty // endpointslice update could be avoid. if slice.Name != sliceToDelete.Name len(slice.Endpoints) != 0 { slice.Name = sliceToDelete.Name slicesToUpdate = append(slicesToUpdate, slice) } slicesToDelete = append(slicesToDelete[:i], slicesToDelete[i+1:]...) } else { i++ } /cc
Hey I'm not sure I completely understand this fix, but thanks for volunteering to take this on! As I recall, this specific section of the reconciler code is part of the "recycling" logic, where an EndpointSlice marked for creation reuses the name of when marked for the deletion, and the operation becomes an update. I'd appreciate a PR if you can also include a test that recreates the buggy behavior + shows that this fixes it.
Thanks for your reply. I will try more test and show this fix does work.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I understand the bug now - it looks the root issue is here: Essentially we'd consider the previous placeholder EndpointSlice no longer desired, mark it for deletion, and then that would mean no EndpointSlices would be left, triggering a new placeholder slice to be marked for creation.

The recycling logic you linked above essentially merges creates and deletes into updates, resulting in the behavior you described of empty EndpointSlices always being updated during the resync period. I think this should be relatively straightforward to fix with some changes to the code I referenced above.
/assign
GitHub didn't allow me to assign the following users: spencerhance. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: glog.V(2).Infof("couldn't find properties for %s, skipping", key) continue } <ins> if details.Type == nil { glog.V(2).Infof("nil details for %s, skipping", key) continue } </ins> fieldType := *details.Type if value == nil { glog.V(2).Infof("Skipping nil field: %s", key)

This happens with several different objects that I tried creating.
happens with simplest pod I can find:
have fix, but working on test.


Question: return "<none>" } <ins> // findNodeRole returns the role of a given node, or "" if none found. // The role is determined by looking in order for: // * a kubernetes.io/role label // * a kubeadm.alpha.kubernetes.io/role label // If no role is found, ("", nil) is returned func findNodeRole(node *api.Node) string { if role := node.Labels[kubeadm.NodeLabelKubeadmAlphaRole]; role != "" { return role } // No role found return "" } </ins> func printNodeList(list *api.NodeList, options printers.PrintOptions) ([]metav1alpha1.TableRow, error) { rows := make([]metav1alpha1.TableRow, 0, len(list.Items)) for i := range list.Items {

used to show masters, but as of it no longer does. This is not in accordance with our deprecation policy I believe. cc
That commit/PR is not relevant as there are no supported users/deployment methods that are using label. kubeadm v1.4 or v1.5 was the last. What you're really looking for is who/what removed printing things based on the label...
No blame on you The regression is that we have stopped printing masters in . There may be more to it that a simple revert of , but we should probably start there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed in recent versions


Question: # API Server using the current Aggregator" RUN go get -d k8s.io/sample-apiserver  && cd ${GOPATH}/src/k8s.io/sample-apiserver  <del> && git checkout --track origin/release-1.7  </del> <ins> && git checkout --track origin/release-1.10  </ins> && CGO_ENABLED=0 GOOS=linux GOARCH=BASEARCH go install . # for arm, go install uses go/bin/linux_arm, so just find the file and copy it to the root so

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: indexer.Add(resourceQuota) // old service was a load balancer, but updated version is a node port. <del> oldService := &api.Service{ ObjectMeta: api.ObjectMeta{Name: "service", Namespace: "test"}, </del> <ins> existingService := &api.Service{ ObjectMeta: api.ObjectMeta{Name: "service", Namespace: "test", ResourceVersion: "1"}, </ins> Spec:       api.ServiceSpec{Type: api.ServiceTypeLoadBalancer}, } newService := &api.Service{

With the migration to podutils and adding some presubmits (from periodic jobs) we are seeing some failures in presubmits. The error seems to have issues finding the right node to create.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig testing
any leads on solving this one?
see notes on slack -
/assign
this looks good now!


Question: // initializeRuntimeDependentModules will initialize internal modules that require the container runtime to be up. func (kl *Kubelet) initializeRuntimeDependentModules() { if err := kl.cadvisor.Start(); err != nil { <del> kl.runtimeState.setInternalError(fmt.Errorf("failed to start cAdvisor %v", err)) </del> <ins> // Fail kubelet and rely on the babysitter to retry starting kubelet. // TODO(random-liu): Add backoff logic in the babysitter glog.Fatalf("Failed to start cAdvisor %v", err) </ins> } // eviction manager must start after cadvisor because it needs to know if the container runtime has a dedicated imagefs if err := kl.evictionManager.Start(kl, kl.getActivePods, evictionMonitoringPeriod); err != nil {

master-informing ci-crio-cgroupv1-node-e2e-conformanceci-crio-cgroupv1-node-e2e-conformance.Overall kubetest.Node Tests E2eNode Suite.[It] [sig-node] Container Runtime blackbox test on terminated container should report termination message from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance] E2eNode Suite.[It] [sig-node] Kubelet when scheduling a busybox command in a pod should print the output to logs [NodeConformance] [Conformance] E2eNode Suite.[It] [sig-node] Kubelet when scheduling a read only busybox container should not write to root filesystem [LinuxOnly] [NodeConformance] [Conformance] E2eNode Suite.[It] [sig-node] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance] ! 2023-05-15 12:08 PDT E2eNode Suite: [It] [sig-node] Pods should support retrieving logs from the container over websockets [NodeConformance] [Conformance] E2eNode Suite: [It] [sig-node] Kubelet when scheduling a read only busybox container should not write to root filesystem [LinuxOnly] [NodeConformance] [Conformance] E2eNode Suite: [It] [sig-node] Container Runtime blackbox test on terminated container should report termination message from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance] E2eNode Suite: [It] [sig-node] Kubelet when scheduling a busybox command in a pod should print the output to logs [NodeConformance] [Conformance] Node Tests No response /sig node cc
~partially blocked on edit: I got the compatibility matrix confused, this is not blocked on that
/reopen Better to close once we see it green
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /triage accepted Head over to the Slack thread if you want to see more discussion:
Working on a fix in
Green tests:

Recently we observe increased number of large-cluster startup failures. After very little digging it seems to be a problem in kubelet/cAdvisor/docker. We're seeing the following log repeat itself until the timeout: P0 as it happened at least 3 time in last 2 days. cc
Those particular logs come from kubelet on master machine in this run:
- wild theory. is it possible that the kube-logrotate.service has to be running before Looking at: If so, we could add a in the for Right?
caused kubelet not being able to sync pods. Could it be possible that kubelet-logrotate get removed while cadvisor was traversing the directory recursively to add all of them? If so, we should ignore such errors since the file doesn't exist anymore. /cc
I didn't see in my cluster. Only
logrotation is triggered by and only runs hourly. I assume it was run while cadvisor was starting up, so cadvisor saw the short-lived cgroup. All my comments above are just speculation :)
Offline discussed with and should be the root cause. In this master, we can see: It means that cadvisor did see during . However, because is only a short lived service, when cadvisor tried to , it has already been deleted. That's why we get the error: I've checked some normal clusters, cadvisor doesn't see at all, because it is too short lived. This is only about timing. :) Anyway, cadvisor shouldn't fail when the cgroup is missing at watching, which sometimes does happen. I'll send a fix soon.
Thanks a lot for this debugging! And waiting for the fix.
FYI, I've sent a PR to fix cadvisor side Once that PR is included in kubernetes, we can close this one. :)
Awesome - thanks
Shouldn't we run into the same issue for 1.3 GKE release? If we do, we need to patch 1.3. cc/


Question: // checkHostPortConflicts detects pods with conflicted host ports. func checkHostPortConflicts(pods []*api.Pod) (fitting []*api.Pod, notFitting []*api.Pod) { <del> ports := map[int]bool{} extract := func(p *api.ContainerPort) int { return p.HostPort } </del> <ins> ports := util.StringSet{} </ins> // Respect the pod creation order when resolving conflicts. sort.Sort(podsByCreationTime(pods)) for _, pod := range pods { <del> if errs := validation.AccumulateUniquePorts(pod.Spec.Containers, ports, extract); len(errs) != 0 { </del> <ins> if errs := validation.AccumulateUniqueHostPorts(pod.Spec.Containers, &ports); len(errs) != 0 { </ins> glog.Errorf("Pod %q: HostPort is already allocated, ignoring: %v", kubecontainer.GetPodFullName(pod), errs) notFitting = append(notFitting, pod) continue

Pods which use to bind both TCP and UDP sockets to the same port number are rejected with the error: An example pod follows.
Looks like we don't check protocol
The issue was discussed before in , which was closed due to it being low priority.
Hmm, that's 2 pods and the scheduler, this is 2 containers in the same pod and validation at create time. Maybe we should just be more clear about where we stand with hostPort.
Posting here what I posted in the other bug: This can't be worked around when running NFS from a pod. Linux does not allow the NFS port (2049) to be changed, and the Portmap port (111) also cannot be changed. Additionally, you can't separate the UDP from the TCP ports.
Why does NFS need host ports? , Russell Harmon wrote:
Fix is in , Tim Hockin wrote:


Question: "//test/e2e/network:all-srcs", "//test/e2e/node:all-srcs", "//test/e2e/perftype:all-srcs", <ins> "//test/e2e/reporters:all-srcs", </ins> "//test/e2e/scheduling:all-srcs", "//test/e2e/servicecatalog:all-srcs", "//test/e2e/storage:all-srcs",

What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin).

I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: t.Fatal("expected to receive an error") } <del> capturedOutput := readStdErr() </del> <ins> klog.Flush() klog.SetOutput(&bytes.Buffer{}) // prevent further writes into buf capturedOutput := buf.String() </ins> // We don't expect stack trace from the panic to be included in the log. if isStackTraceLoggedByRuntime(capturedOutput) {

/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?


Question: // It supports wrapped errors and returns StatusReasonUnknown when // the error is nil or doesn't have a status. func ReasonForError(err error) metav1.StatusReason { <del> if status := APIStatus(nil); errors.As(err, &status) { </del> <ins> if status, ok := err.(APIStatus); ok || errors.As(err, &status) { </ins> return status.Status().Reason } return metav1.StatusReasonUnknown } func reasonAndCodeForError(err error) (metav1.StatusReason, int32) { <del> if status := APIStatus(nil); errors.As(err, &status) { </del> <ins> if status, ok := err.(APIStatus); ok || errors.As(err, &status) { </ins> return status.Status().Reason, status.Status().Code } return metav1.StatusReasonUnknown, 0

I have a controller that creates objects. It uses to notice when create fails due to a deleting namespace. This check was working well until I moved it away from each call. In some cases, the error is wrapped using causing the function to always return false. I expected to work on both wrapped and unwrapped errors, like the functions do. This short program shows the issue: I am able to work around it by first calling . This type assertion should be replaced with a call to : /sig api-machinery v1.22.5, but I also see the affected code in master. <details</detailsn/a n/a k3s n/a n/a
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: map[string]string{ "type": "pd-standard", }, <del> "1Gi", "1Gi", </del> <ins> "1G", "1G", </ins> func(volume *v1.PersistentVolume) error { return checkGCEPD(volume, "pd-standard") },

GCE PDs are allocated in chunks of GBs (not GiB) but function incorrectly creates volume in chunks of GiB - cc /sig storage
/kind bug
/assign
/priority important-soon
Hey I submit a pr for this issue. Your suggestions are very important. Thanks.


Question: usage := "Filename, directory, or URL to file that contains the configuration to apply" kubectl.AddJsonFilenameFlag(cmd, &options.Filenames, usage) cmd.MarkFlagRequired("filename") <ins> cmd.Flags().Bool("overwrite", true, "Automatically resolve conflicts between the modified and live configuration by using values from the modified configuration") </ins> cmdutil.AddValidateFlags(cmd) cmdutil.AddRecursiveFlag(cmd, &options.Recursive) cmdutil.AddOutputFlagsForMutation(cmd)

This issue tracks the implementation of the --overwrite flag in kubectl apply, per in .
Changed the title to address only the --overwrite flag, since deals with the --force flag.
progress on this issue is blocked waiting for to merge.


Question: glog.Errorf("Failed to start ImageManager, images may not be garbage collected: %v", err) } <ins> err = kl.cadvisor.Start() if err != nil { kl.recorder.Eventf(kl.nodeRef, "kubeletSetupFailed", "Failed to start CAdvisor %v", err) glog.Errorf("Failed to start CAdvisor, system may not be properly monitored: %v", err) } </ins> err = kl.containerManager.Start() if err != nil { kl.recorder.Eventf(kl.nodeRef, "kubeletSetupFailed", "Failed to start ContainerManager %v", err)

This is blocking (reboot test). In running a test that reboots all of the nodes and then runs the e2e tests, we saw a failure with the influxdb / heapster monitoring test: The heapster logs, from running (), shows a ton of the following kubelet errors for all containers (this one happens to be for skydns): The shows a bunch of 500 errors, such as: However, the apiserver thinks that all the pods are running ( gives this ) Docker thinks that the pods are running ( on that node gives this ) In summary: apiserver and docker think containers are running but kubelet can't find them this manifested in the influxdb / heapster monitoring e2e test after node reboot I've still got the cluster around if anyone would like to debug. +cc (helped me debug this (thanks!)) +cc (node) +cc (interested in getting in, which this is blocking)
Can I access your test nodes?
Is this an active problem? I'd check if the cAdvisor thinks the containers exist (it is the one returning the error). I'd then scan the logs to see if cAdvisor had any errors related to that container.
Unfortunately during debugging, I restart kubelet service on that node, the issue is gone. The problem could be cadvisor cannot get stats, but I couldn't find any cadvisor related error messages. How easy to reproduce the issue here? Next time we might access cadvisor's page to see if there is no such container information. Thanks!
(and Thanks for looking into this! I believe I can reproduce easily but I'll confirm. Where do I look for cAdvisor info? I don't see a log in or , and I don't see a cAdvisor container running?
cAdvisor is running as part of kubelet now. If you can reproduce the issue, could you please change kubelet's log level to 4 so that we have more cadvisor log here? Thanks!
Yes! How do I do that?

I can reproduce this by doing the following: After this, I got two failures: ... and I have another node with more of the same kubelet errors: (If I need to change the Kubelet log level at compile-time I can do that and repro on a new cluster; otherwise, I've got another cluster with the issue that can be introspected...)
There is a race between cadvisor initialization and docker. cadvisor failed to register docker factory, and bailed out. is going to make cadvisor more resilient to the issue of docker is not available, and have some retry logic. I am going to change kubelet initialization logic a little bit to make sure docker is running before we start cadvisor.
Awesome, thank you Dawn! Are there issues or PRs I can follow to know when this is unblocked?
is one of them. is working on the second.


Question: ], ) <ins> go_test( name = "go_default_test", srcs = ["core_test.go"], embed = [":go_default_library"], deps = [ "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/client-go/discovery:go_default_library", "//staging/src/k8s.io/client-go/discovery/fake:go_default_library", "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", ], ) </ins> filegroup( name = "package-srcs", srcs = glob(["**"]),

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } } <del> // given an existing endpoint slice and no pods matching the service, the existing // slice should be updated to a placeholder (not deleted) </del> <ins> // given an existing placeholder endpoint slice and no pods matching the service, the existing // slice should not change the placeholder </ins> func TestReconcile1EndpointSlice(t *testing.T) { <del> client := newClientset() setupMetrics() </del> namespace := "test" <del> svc, endpointMeta := newServiceAndEndpointMeta("foo", namespace) endpointSlice1 := newEmptyEndpointSlice(1, namespace, endpointMeta, svc) </del> <ins> svc, epMeta := newServiceAndEndpointMeta("foo", namespace) emptySlice := newEmptyEndpointSlice(1, namespace, epMeta, svc) emptySlice.ObjectMeta.Labels = map[string]string{"bar": "baz"} </ins> <del> _, createErr := client.DiscoveryV1().EndpointSlices(namespace).Create(context.TODO(), endpointSlice1, metav1.CreateOptions{}) assert.Nil(t, createErr, "Expected no error creating endpoint slice") </del> <ins> testCases := []struct { desc        string existing    *discovery.EndpointSlice wantUpdate  bool wantMetrics expectedMetrics }{ { desc:        "No existing placeholder", wantUpdate:  true, wantMetrics: expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 1, numUpdated: 0, numDeleted: 0, slicesChangedPerSync: 1}, }, { desc:        "Existing placeholder that's the same", existing:    newEndpointSlice(&svc, &endpointMeta{Ports: []discovery.EndpointPort{}, AddressType: discovery.AddressTypeIPv4}), wantMetrics: expectedMetrics{desiredSlices: 0, actualSlices: 0, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 0, numUpdated: 0, numDeleted: 0, slicesChangedPerSync: 0}, }, { desc:        "Existing placeholder that's different", existing:    emptySlice, wantUpdate:  true, wantMetrics: expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 0, numUpdated: 1, numDeleted: 0, slicesChangedPerSync: 1}, }, } </ins> <del> numActionsBefore := len(client.Actions()) r := newReconciler(client, []*corev1.Node{{ObjectMeta: metav1.ObjectMeta{Name: "node-1"}}}, defaultMaxEndpointsPerSlice) reconcileHelper(t, r, &svc, []*corev1.Pod{}, []*discovery.EndpointSlice{endpointSlice1}, time.Now()) assert.Len(t, client.Actions(), numActionsBefore+1, "Expected 1 additional clientset action") actions := client.Actions() assert.True(t, actions[numActionsBefore].Matches("update", "endpointslices"), "Action should be update endpoint slice") </del> <ins> for _, tc := range testCases { t.Run(tc.desc, func(t *testing.T) { client := newClientset() setupMetrics() </ins> <del> slices := fetchEndpointSlices(t, client, namespace) assert.Len(t, slices, 1, "Expected 1 endpoint slices") </del> <ins> existingSlices := []*discovery.EndpointSlice{} if tc.existing != nil { existingSlices = append(existingSlices, tc.existing) _, createErr := client.DiscoveryV1().EndpointSlices(namespace).Create(context.TODO(), tc.existing, metav1.CreateOptions{}) assert.Nil(t, createErr, "Expected no error creating endpoint slice") } </ins> <del> assert.Regexp(t, "^"+svc.Name, slices[0].Name) assert.Equal(t, svc.Name, slices[0].Labels[discovery.LabelServiceName]) assert.EqualValues(t, []discovery.EndpointPort{}, slices[0].Ports) assert.EqualValues(t, []discovery.Endpoint{}, slices[0].Endpoints) expectTrackedGeneration(t, r.endpointSliceTracker, &slices[0], 1) expectMetrics(t, expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 0, numUpdated: 1, numDeleted: 0, slicesChangedPerSync: 1}) </del> <ins> numActionsBefore := len(client.Actions()) r := newReconciler(client, []*corev1.Node{{ObjectMeta: metav1.ObjectMeta{Name: "node-1"}}}, defaultMaxEndpointsPerSlice) reconcileHelper(t, r, &svc, []*corev1.Pod{}, existingSlices, time.Now()) var numUpdates int if tc.wantUpdate { numUpdates = 1 } wantActions := numActionsBefore + numUpdates assert.Len(t, client.Actions(), wantActions, "Expected %d additional clientset actions", numUpdates) slices := fetchEndpointSlices(t, client, namespace) assert.Len(t, slices, 1, "Expected 1 endpoint slices") if !tc.wantUpdate { assert.Regexp(t, "^"+svc.Name, slices[0].Name) assert.Equal(t, svc.Name, slices[0].Labels[discovery.LabelServiceName]) assert.EqualValues(t, []discovery.EndpointPort{}, slices[0].Ports) assert.EqualValues(t, []discovery.Endpoint{}, slices[0].Endpoints) if tc.existing == nil { expectTrackedGeneration(t, r.endpointSliceTracker, &slices[0], 1) } } expectMetrics(t, tc.wantMetrics) }) } } func TestPlaceHolderSliceCompare(t *testing.T) { testCases := []struct { desc string x    *discovery.EndpointSlice y    *discovery.EndpointSlice want bool }{ { desc: "Both nil", want: true, }, { desc: "Y is nil", x:    &discovery.EndpointSlice{}, want: false, }, { desc: "X is nil", y:    &discovery.EndpointSlice{}, want: false, }, { desc: "Both are empty and non-nil", x:    &discovery.EndpointSlice{}, y:    &discovery.EndpointSlice{}, want: true, }, { desc: "Only ObjectMeta.Name has diff", x: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Name: "foo", }}, y: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Name: "bar", }}, want: true, }, { desc: "Only ObjectMeta.Labels has diff", x: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ "foo": "true", }, }}, y: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ "bar": "true", }, }}, want: false, }, { desc: "Creation time is different", x: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ CreationTimestamp: metav1.Unix(1, 0), }}, y: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ CreationTimestamp: metav1.Unix(2, 0), }}, want: true, }, { desc: "Different except for ObjectMeta", x:    &discovery.EndpointSlice{AddressType: discovery.AddressTypeIPv4}, y:    &discovery.EndpointSlice{AddressType: discovery.AddressTypeIPv6}, want: false, }, } for _, tc := range testCases { t.Run(tc.desc, func(t *testing.T) { got := placeholderSliceCompare.DeepEqual(tc.x, tc.y) if got != tc.want { t.Errorf("sliceEqual(%v, %v) = %t, want %t", tc.x, tc.y, got, tc.want) } }) } </ins> } // when a Service has PublishNotReadyAddresses set to true, corresponding

My cluster contains about 4w endpointslices. There are 3 related metrics (endpointslicecontrollerendpointsaddedpersynccount, endpointslicecontrollerendpointsremovedpersynccount, endpointslicecontrollerchanges) to indicates endpoint add remove and update. I found that the apiserver periodically receives lots of endpointslice update requests. In terms of the number of requests, the endpointslice has been fully updated. But from the endpointslice resource itself, it seems that it has not been updated, because "" annotation didn't change It seems that there is no need to update the endpointslice with no really changes. On the other hand, a large number of ep slice update requests will stress the apiserver. In order to make the phenomenon more obvious, the min-resync-period parameter of controller-manager can be adjusted to 1 minute. Create one or more empty endpointslices, and then you can find in the apiserver logs that requests to update these endpointslice every one to two minutes. For clusters with lots of empty endpointslices, this bug may affect the stability of it. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
there were different improvements in this area, but since 1.20 is going EOL in 8 days it will be good to know if this happens in 1.21 /cc
I'm glad to recieve your comment. I'm not sure because I only have cluster with version 1.20.10. Expect someone who could provide more detail about it. It is actually happening in my cluster, but I don't know how it happens.
I've seen this in 1.20 clusters as well, but have not seen it in 1.21.

My theory was that this is mostly or entirely fixed by Prior to that, any change to EndpointSlice labels would not trigger an EndpointSlice controller sync loop, but the resync period would result in those EndpointSlices being updated at that point. more recent fix () seems like it could also be relevant, but I don't think it matches what's described here. In that case, I believe we're unnecessarily updating EndpointSlices if something irrelevant in the Pod changes (resource version), but I don't believe there's anything special there that would mean we weren't updating EndpointSlices until the resync period. So with all of that said, I believe this is fixed in 1.21+, but let me know if anyone's able to recreate this in those versions and/or if is actually more relevant than it appears on the surface.
Thanks for your reply I will check it.
I confirmed that the periodic updates in my cluster are all empty endpointslices. There are l lot of IPv6 endpointslice in my cluster. But it shouldn't be updated when endpointslice-controller couldn't list pods by service label selector.
/assign
I found that when endpointslice's endpoint array is empty, endpointslice-controller updates it every periodic resync, although it didn't change. I consider it a bug and would like to fork and fix it. The code below could fix the issue of updating empty endpointslices, but I'm not sure if it will introduce new issues. for i := 0; i < len(slicesToDelete); { if len(slicesToCreate) == 0 { break } sliceToDelete := slicesToDelete[i] slice := slicesToCreate[len(slicesToCreate)-1] // Only update EndpointSlices that are owned by this Service and have // the same AddressType. We need to avoid updating EndpointSlices that // are being garbage collected for an old Service with the same name. // The AddressType field is immutable. Since Services also consider // IPFamily immutable, the only case where this should matter will be // the migration from IP to IPv4 and IPv6 AddressTypes, where there's a // chance EndpointSlices with an IP AddressType would otherwise be // updated to IPv4 or IPv6 without this check.

if sliceToDelete.AddressType == slice.AddressType && ownedBy(sliceToDelete, service) { slicesToCreate = slicesToCreate[:len(slicesToCreate)-1] // We must consider the situation that an empty unchanged endpointslice no need to update. If the one to be deleted and the one to be created are the same endpointslice which is empty, skip the update step. // eg: If there are many headless svc with single stack ipv4 for example, a lot of ipv6 empty // endpointslice update could be avoid. if slice.Name != sliceToDelete.Name len(slice.Endpoints) != 0 { slice.Name = sliceToDelete.Name slicesToUpdate = append(slicesToUpdate, slice) } slicesToDelete = append(slicesToDelete[:i], slicesToDelete[i+1:]...) } else { i++ } /cc
Hey I'm not sure I completely understand this fix, but thanks for volunteering to take this on! As I recall, this specific section of the reconciler code is part of the "recycling" logic, where an EndpointSlice marked for creation reuses the name of when marked for the deletion, and the operation becomes an update. I'd appreciate a PR if you can also include a test that recreates the buggy behavior + shows that this fixes it.
Thanks for your reply. I will try more test and show this fix does work.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I understand the bug now - it looks the root issue is here: Essentially we'd consider the previous placeholder EndpointSlice no longer desired, mark it for deletion, and then that would mean no EndpointSlices would be left, triggering a new placeholder slice to be marked for creation.

The recycling logic you linked above essentially merges creates and deletes into updates, resulting in the behavior you described of empty EndpointSlices always being updated during the resync period. I think this should be relatively straightforward to fix with some changes to the code I referenced above.
/assign
GitHub didn't allow me to assign the following users: spencerhance. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: //modify annotations pod.Annotations["builder"] = "foo" <ins> pod.ResourceVersion = "" // to force update </ins> _, err = f.Client.Pods(f.Namespace.Name).Update(pod) Expect(err).NotTo(HaveOccurred())

In parallel suite:
Can you recreate this? It could be a race or a buffer not flushing.
I'm having a look
Anoter example of the failure: - awesome thanks!
unluckily I cannot seet this page. Thanks anyway
to delegate? (If you don't think it's a node bug, maybe can find an owner?)
Here's our latest conformance log from master with the labels/annotations both failing (mesos/docker cluster): Downward API volume should update labels on modification [Conformance] Downward API volume should update annotations on modification [Conformance]
These failures can be reproduced on master with a single command:
thanks for the logs. I'm investigating the issue using (never worked with so I'm not sure if it's the same or not) Running only one instance of like this --host="127.0.0.1:8080" --provider="local" --ginkgo.v=true --"*on.modification" --kubeconfig="$" basically your command (I guess) I never had a failure. Running the same command in a loop it's different and I managed to reproduce the issue, most of the time the timeout looks to aggressive 15s (while in most of the tests the timeout is something like 5 minutes). since it's seems that this is blocking for several person I'd like to raise the timout (at least to 30seconds). I'm not saying that this is the fix (the old joke about ) but at least it should reduce the problem on busy system.
It looks to me that everytime the fails happens I ran into something like: the same is valid for logs. I tried to in the code (maybe I'm blind) but I couldn't find... Any ideas?
- those lines are coming from this metric: the "kubelet_" at the beginning is "subsystem" in prometheus metrics, that's why you probably didn'g find it
Actually it's printed here:
thanks thanks
We should verify if increasing the timeout: fixed this issue.
Still fails on the mesos/docker cluster on master:
I have been informed that the mesos executor wasn't actually handling annotation/label updates. So these tests were failing legitimately on the mesos/docker cluster.

This will hopefully be fixed by
don't know if makes sense for you but kubelet broke this and we add to fix
Another failure:
:( Sorry I didn't see this... Ok. Having a look again (but today I'm pretty busy)
is not enough. We need also to increase timeout in image which generates the logs sorry for this I didn't realize it when we talked about this thanks for the file!
submitted to increase timout in running image
- great - thanks for looking into it!
I'm reopening it, so that we can monitor if this really fixed the problem. Will close it if we don't observe more errors for a couple of days.
There weren't any failures of this test within last 3 days - closing! Thanks
Saw this flake again in a GKE run:
same test but another issue. we got this: . While the previous issue was a too short timeout (container exited too early for slow envs). would you like to open another issue for this?
Working on this to propose a fix
submitted please have a look
I saw you xref this issue from . I cannot see the relations (and I could be wrong). Mind explain? Let me know if I need to Thanks
- the test you are trying to fix failed in the presubmit of that PR: I didn't review but it's not merged yet, so we don't know if that helps (hopefully yes :))
I think will help on these kind of issues (retrying on the update). thanks for the explanation.
I encountered a failure of this test for a different reason, I can create a separate issue if you think it's unrelated: link:
In the current implementation is performed without retry. implement a retry mechanism similar to
thanks. FYI, Actually we can set the Pod.ResourceVersion="", that will force an update, it would be cleaner than using a Poll.
I'm going to update the PR with what you're suggesting. Thanks


Question: **Fixed Versions**: - kube-apiserver v1.25.4 - kube-apiserver v1.24.8 <del> - kube-apiserver v1.23.13 </del> <ins> - kube-apiserver v1.23.14 </ins> - kube-apiserver v1.22.16 This vulnerability was reported by Yuval Avrahami of Palo Alto Networks

The section of the Changelog for 1.23.14 (and sibling bugfix releases) says that 1.23.13 both has the vuln and also has the fix. The Fix version should be 1.23.14 in the above and sibling changelogs. Load the link above. No response N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
cc
For v1.23.14, the title is . This seems to be valid. means
You're looking at the wrong text. Follow the link and see .13 in both lists: kube-apiserver v1.25.0 - v1.25.3 kube-apiserver v1.24.0 - v1.24.7 kube-apiserver v1.23.0 - v1.23.13 kube-apiserver v1.22.0 - v1.22.15 kube-apiserver <= v1.21.? kube-apiserver v1.25.4 kube-apiserver v1.24.8 kube-apiserver v1.23.13 kube-apiserver v1.22.16
Got it. Thanks for pointing out the location.
I opened to fix it.
/reopen Can you please cherry-pick to the release branches (1.25-1.22)?
Reopened this issue. <detailsIn response to : /reopen Can you please cherry-pick to the release branches (1.25-1.22)? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Of course. Done
Cherry-picks are done, thanks a lot /close
Closing this issue. <detailsIn response to : Cherry-picks are done, thanks a lot /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return true, nil } else if os.IsNotExist(err) { return false, nil <del> } else if isCorruptedMnt(err) { </del> <ins> } else if IsCorruptedMnt(err) { </ins> return true, err } else { return false, err } } <del> // isCorruptedMnt return true if err is about corrupted mount point func isCorruptedMnt(err error) bool { </del> <ins> // IsCorruptedMnt return true if err is about corrupted mount point func IsCorruptedMnt(err error) bool { </ins> if err == nil { return false }

/kind bug What happened: Kubelet failed to unmount volumes if the backend ISCSI volume is not available. This also applies to other network attached volumes. What you expected to happen: Kubelet should be able to unmount the volumes. A manual unmount works even when the backend network volume is not available. How to reproduce it (as minimally and precisely as possible): Mount an ISCSI/network volume and disconnect/de-provision ISCSI target. Anything else we need to know?: XFS shuts down the filesystem due to I/O errors. Kubelet uses "stat" calls to figure out if it is a valid mount point or it is already unmounted. Stat calls fail with EIO and Kubelet never proceeds to unmount the volume. Environment: Kubernetes version (use ): Kubernetes 1.10.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7.4 Kernel (e.g. ): Linux appserv37 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Manual bring up Others:
/sig storage
IsLikelyNotMountPoint is using stat call to figure the filesystem is mounted or not. Stat calls are failing with EIO when there are I/O errors on the backend and Kubelet never proceeds to unmount the filesystem. Also, isLikelyNotMountPoint is not looking for "isCorruptedMnt" errors. All other calls have it.
looks like the nfs plugin is calling . Can iscsi use that?
didn't include error "Input/output error" () We should add it. I make a fix, PTAL, Thank you!
Thanks. Trying out the change. Will keep you posted if we need any other changes.
Stock ISCSI driver is working fine with EIO change. But more changes are required for making it work with Flexvolume drivers. I have the change list. I can club your change in or let me know if you want to go ahead and push this fix. I will push another PR to fix flexvolume issues.
Please club my change, I will close my PR. Thanks!


Question: package main import ( <del> "io/ioutil" "os" </del> <ins> "io/ioutil" "os" </ins> <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/kubectl/cmd" </del> <ins> "k8s.io/kubernetes/pkg/kubectl/cmd" "k8s.io/kubernetes/pkg/kubectl/cmd/util" </ins> ) func main() { <del> kubectl := cmd.NewFactory(nil).NewKubectlCommand(os.Stdin, ioutil.Discard, ioutil.Discard) kubectl.GenBashCompletionFile("out.sh") </del> <ins> kubectl := cmd.NewKubectlCommand(util.NewFactory(nil), os.Stdin, ioutil.Discard, ioutil.Discard) kubectl.GenBashCompletionFile("out.sh") </ins> } ```

BUG REPORT: /kind bug What happened: Arguments to a command inside pod are treated as kubectl flags and deprecation warnings are shown for them from . What you expected to happen: Arguments to a command inside pod should not be treated as kubectl flags by . How to reproduce it: The arguments , , , passed to the command inside the pod above are treated as flags and deprecation warnings are shown for them from . Environment: Kubernetes version: 9: Cloud provider or hardware configuration: Ubuntu Xenial VM. OS:Kernel:Install tools: minikube:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig cli Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is it ok if we remove this warning ?
the warning should be removed just for the arguments to a command inside pod that are passed after the , and not for the arguments to itself. The warning is still valid and needed for arguments with underscore () to itself.
This is a bug of cobra. I've sent out a PR to fix it. And bumped to the fixed version in .
Thanks for fixing this!


Question: "IngressRule.HTTP.Paths.Backend.Service v1beta1": { input: v1beta1, output: `Name:             bar <ins> Labels:           id1=app1 id2=app2 </ins> Namespace:        foo Address: Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)

When command 'kubectl describe ing' Name: test-ing Namespace: practice Address: 10.10.01.10 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found) TLS: test terminates Rules: Host Path Backends ---- ---- -------- / test-svc:80 (<none) Annotations: nginx Events: <noneResult has no information about labels. But Pod, Deployment, Service resources has label information like below. Name: test-room-test-mjsbz Namespace: test Priority: 0 Node: aks-test-test-test/22.22.222. Start Time: Fri, 09 Jul 2021 05:45:26 +0000 Labels: app=test pod-template-hash=test role=mediatest Why ingress has no information about labels? Actually when delete ingresss with labels is working correctly. Kubernetes version (use ): 1.18.14 Cloud provider or hardware configuration: AKS (Azure) OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig
/sig
/sig cli
/assign
/assign
/triage accepted /remove-kind bug /kind feature


Question: Expect(err).NotTo(HaveOccurred()) dirName := strings.Fields(res)[0] hostDir := "/mnt/disks/by-uuid/google-local-ssds-scsi-fs/" + dirName <del> return generateLocalTestVolume(hostDir, config, GCELocalSSDVolumeType, node) </del> <ins> // gce local ssd does not need to create a directory return &localTestVolume{ node:            node, hostDir:         hostDir, localVolumeType: GCELocalSSDVolumeType, } </ins> } func setupLocalVolumeDirectory(config *localTestConfig, node *v1.Node) *localTestVolume {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Since 8/31, these tests have started flaking, and then started failing consistently. https://k8s- https://k8s- /assign
/kind failing-test
/milestone v1.12
Does this need cherry picked back to 1.11? Still have upgrade failures with this symptom, eg: https://k8s-
No, the localssd tests are not failing in the upgrade suite
/reopen I'm going to call this one a partial fix, as we are still seeing the "Set fsGroup for local volume should not set different fsGroups for two pods simultaneously" error on a whole bunch of test suites.
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you link me to other suites where it is failing?
okay "a whole bunch" was exaggerated, I apologize: https://k8s- https://k8s-
The upgrade tests are not failing with "gce-localssd", which is what this issue is addressing, so is not related. The upgrade tests issues are being tracked in /close
Closing this issue. <detailsIn response to : The upgrade tests are not failing with "gce-localssd", which is what this issue is addressing, so is not related. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> // SetUpSampleAPIServer sets up a sample-apiserver. // // Important: // Test cases that call this function should be marked as serial due to potential conflicts // with other test cases that also set up a sample-apiserver. For more information, see // https://github.com/kubernetes/kubernetes/issues/119582#issuecomment-2215054411. </ins> func SetUpSampleAPIServer(ctx context.Context, f *framework.Framework, aggrclient *aggregatorclient.Clientset, image string, n sampleAPIServerObjectNames, apiServiceGroupName, apiServiceVersion string) { ginkgo.By("Registering the sample API server.") client := f.ClientSet

e2e conformance jobs, seen in aws, capz-windows, and kind seen back to 1.28 shows flakes going back as far as history exists No response [FAILED] could not find group version resource for dynamic client and wardle/flunders (discovery error: <nil, discovery results: map[schema.GroupVersionResource]struct {}{schema.GroupVersionResource{Group:"", Version:"v1", Resource:"bindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"configmaps"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpoints"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"limitranges"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"persistentvolumeclaims"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"pods"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"podtemplates"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"replicationcontrollers"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"resourcequotas"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"secrets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"serviceaccounts"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"services"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"controllerrevisions"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"daemonsets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"deployments"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"replicasets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"statefulsets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1",

Resource:"localsubjectaccessreviews"}:struct {}{}, schema.GroupVersionResource{Group:"autoscaling", Version:"v2", Resource:"horizontalpodautoscalers"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"cronjobs"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"jobs"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"leases"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpointslices"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"ingresses"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"networkpolicies"}:struct {}{}, schema.GroupVersionResource{Group:"policy", Version:"v1", Resource:"poddisruptionbudgets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"rolebindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"roles"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"csistoragecapacities"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1beta1", Resource:"flunders"}:struct {}{}}) or - [FAILED] could not find group version resource for dynamic client and wardle/flunders (discovery error: unable to retrieve the complete list of server APIs: stale GroupVersion discovery: , discovery results: map[schema.GroupVersionResource]struct {}{schema.GroupVersionResource{Group:"", Version:"v1", Resource:"bindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"configmaps"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpoints"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"limitranges"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1",

Resource:"persistentvolumeclaims"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"pods"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"podtemplates"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"replicationcontrollers"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"resourcequotas"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"secrets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"serviceaccounts"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"services"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"controllerrevisions"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"daemonsets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"deployments"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"replicasets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"statefulsets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"localsubjectaccessreviews"}:struct {}{}, schema.GroupVersionResource{Group:"autoscaling", Version:"v2", Resource:"horizontalpodautoscalers"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"cronjobs"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"jobs"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"leases"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpointslices"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"ingresses"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"networkpolicies"}:struct {}{}, schema.

GroupVersionResource{Group:"policy", Version:"v1", Resource:"poddisruptionbudgets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"rolebindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"roles"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"csistoragecapacities"}:struct {}{}}) Unclear if the issue is discovery going stale, or the aggregated server not responding, or something else /sig api-machinery
last flake on sig-release-master-blocking testgrid is 03-05 As it does not flake on the board anymore and still flake in other CIs, remove it from release signal board.
/assign Based on , I don't see it flaking as far as history exists. It seemed to have started flaking around March/April 2023. This also corresponds to when aggregated discovery was enabled by default so it may be related. Will do some more digging.
/triage accepted
Hey Is this issue fixed? If yes, from which version of kubernetes will this fix be available from?
This still flakes in some CIs. The flake rate is very low, but it is a conformance test.
Thanks for the quick response. I see it is very flaky. Is there any active work going on to fix this? If yes, is there any estimated time by which it could be fixed?
do you have some update on this?
cc.

I see the flake in recent run of later find it flake in as well. (not flake now) or More flakes can be found in /kind flake <!-- If this is a flake, please add: /kind flake --/sig api-machinery a slack thread a similar issue: for windows CI
/cc
Likely a test job misconfiguration. /assign /triage accepted
It seems there was a previous related issue
/cc
From Cloud provider testgrid is testing against master k8s branch, while the test requires a change in kube-aggregator, which is not yet cherry-picked to cloud provider testgrid release-1.28
It not only flakes on windows CI.
hit in
Flake in as well.
/remove-sig windows
Flake in .
See
/unassign
this test is still flaking but rarely? from a kind job today
This is still flaking, see a recent duplicate:
Hello! This Flake has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting 02:00 UTC Wednesday 10th July 2024 (about 4 weeks from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this is tagged for 1.31, is it still planned for this release?
/milestone v1.31
It seems it is failing in
cc. Hit the same in the presubmit which ran against one of my PRs. Did some digging into this. It turned out to be a conflict between the following test cases running in parallel: [It] [sig-api-machinery] Aggregator Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] [It] [sig-api-machinery] OpenAPIV3 should contain OpenAPI V3 for Aggregated APIServer Both test cases call . As a result, two cluster-scoped s are created, one for and one for . The existence of the causes to fail, as it results in returning instead of the expected . I also managed to reproduce this issue with hydrophone: See for test logs. Finally, the solution is to mark both test cases as [Serial]. I'll submit a PR shortly for this.

/assign
can we avoid to make them serial? do we have a way to make two tests to not able to run at the same time?
the only way, currently, is to mark (at least) one as . I have plans (more of a long-term vision, really) for more but for now your best recourse is to use . I suppose you could also leverage Kubernetes and have both tests get/remove some sort of agreed upon mutex in the cluster and hang out and wait until the mutex is cleared before running. This would allow them to run in parallel with other specs but would force them to serialize wrt each other if they ever try to run concurrently. While doable, this will introduce a fair bit of complexity and a very different new internal contract into the test suite. It's Almost Certainly a Bad Idea.
I agree that marking the tests as serial is currently our best option. We talked about running all conformance tests with a single invocation. That can be done with without skipping the serial tests: Ginkgo will run non-serial tests in parallel and then the serial ones.
let's mark the non-conformance as serial then, Serial are skipped by default in a lot of places and missing coverage in a conformance test is something I would prefer to avoid


Question: manager.RegisterPod(podWithSecrets("ns1", "other-name", s2)) manager.UnregisterPod(podWithSecrets("ns1", "other-name", s2)) <ins> s5 := secretsToAttach{ containerEnvSecrets: []envSecrets{ {envVarNames: []string{"s7"}}, {envFromNames: []string{"s70"}}, }, } // Check the no-op update scenario manager.RegisterPod(podWithSecrets("ns1", "noop-pod", s5)) manager.RegisterPod(podWithSecrets("ns1", "noop-pod", s5)) </ins> // Now we have: 3 pods with s1, 2 pods with s2 and 2 pods with s3, 0 pods with s4. <del> verify := func(ns, name string, count int) bool { </del> <ins> refs := func(ns, name string) int { </ins> store.lock.Lock() defer store.lock.Unlock() item, ok := store.items[objectKey{ns, name}] if !ok { <del> return count == 0 </del> <ins> return 0 </ins> } <del> return item.refCount == count } assert.True(t, verify("ns1", "s1", 3)) assert.True(t, verify("ns1", "s10", 1)) assert.True(t, verify("ns1", "s2", 3)) assert.True(t, verify("ns1", "s3", 3)) assert.True(t, verify("ns1", "s30", 2)) assert.True(t, verify("ns1", "s4", 2)) assert.True(t, verify("ns1", "s5", 4)) assert.True(t, verify("ns1", "s50", 2)) assert.True(t, verify("ns1", "s6", 0)) assert.True(t, verify("ns1", "s60", 0)) assert.True(t, verify("ns1", "s7", 0)) </del> <ins> return item.refCount } assert.Equal(t, 3, refs("ns1", "s1")) assert.Equal(t, 1, refs("ns1", "s10")) assert.Equal(t, 3, refs("ns1", "s2")) assert.Equal(t, 3, refs("ns1", "s3")) assert.Equal(t, 2, refs("ns1", "s30")) assert.Equal(t, 2, refs("ns1", "s4")) assert.Equal(t, 4, refs("ns1", "s5")) assert.Equal(t, 2, refs("ns1", "s50")) assert.Equal(t, 0, refs("ns1", "s6")) assert.Equal(t, 0, refs("ns1", "s60")) assert.Equal(t, 1, refs("ns1", "s7")) assert.Equal(t, 1, refs("ns1", "s70")) </ins> } func TestCachingSecretManager(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.