Question: assert.NotEqual(t, expectedIPTables, fp.iptablesData.String()) } <ins> func TestProxierDeleteNodePortStaleUDP(t *testing.T) { fcmd := fakeexec.FakeCmd{} fexec := fakeexec.FakeExec{ LookPathFunc: func(cmd string) (string, error) { return cmd, nil }, } execFunc := func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) } cmdOutput := "1 flow entries have been deleted" cmdFunc := func() ([]byte, []byte, error) { return []byte(cmdOutput), nil, nil } // Delete ClusterIP entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) // Delete NodePort entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) ipt := iptablestest.NewFake() fp := NewFakeProxier(ipt, false) fp.exec = &fexec svcIP := "10.20.30.41" svcPort := 80 nodePort := 31201 svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80", Protocol:       v1.ProtocolUDP, } makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *v1.Service) { svc.Spec.ClusterIP = svcIP svc.Spec.Ports = []v1.ServicePort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: v1.ProtocolUDP, NodePort: int32(nodePort), }} }), ) makeEndpointsMap(fp) fp.syncProxyRules() if fexec.CommandCalls != 0 { t.Fatalf("Created service without endpoints must not clear conntrack entries") } epIP := "10.180.0.1" makeEndpointsMap(fp, makeTestEndpoints(svcPortName.Namespace, svcPortName.Name, func(ept *v1.Endpoints) { ept.Subsets = []v1.EndpointSubset{{ Addresses: []v1.EndpointAddress{{ IP: epIP, }}, Ports: []v1.EndpointPort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: v1.ProtocolUDP, }}, }} }), ) fp.syncProxyRules() if fexec.CommandCalls != 2 { t.Fatalf("Updated UDP service with new endpoints must clear UDP entries") } // Delete ClusterIP Conntrack entries expectCommand := fmt.Sprintf("conntrack -D --orig-dst %s -p %s", svcIP, strings.ToLower(string((v1.ProtocolUDP)))) actualCommand := strings.Join(fcmd.CombinedOutputLog[0], " ") if actualCommand != expectCommand { t.Errorf("Expected command: %s, but executed %s", expectCommand, actualCommand) } // Delete NodePort Conntrack entrie expectCommand = fmt.Sprintf("conntrack -D -p %s --dport %d", strings.ToLower(string((v1.ProtocolUDP))), nodePort) actualCommand = strings.Join(fcmd.CombinedOutputLog[1], " ") if actualCommand != expectCommand { t.Errorf("Expected command: %s, but executed %s", expectCommand, actualCommand) } } </ins> // TODO(thockin): add *more* tests for syncProxyRules() or break it down further and test the pieces.

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: method:        "PUT", expectAccept:  false, }, <ins> { name:          "test23", acceptPaths:   DefaultPathAcceptRE, rejectPaths:   DefaultPathRejectRE, acceptHosts:   DefaultHostAcceptRE, rejectMethods: DefaultMethodRejectRE, path:          "/api/v1/namespaces/default/pods/somepod/exec", host:          "localhost", method:        "POST", expectAccept:  false, }, { name:          "test24", acceptPaths:   DefaultPathAcceptRE, rejectPaths:   "", acceptHosts:   DefaultHostAcceptRE, rejectMethods: DefaultMethodRejectRE, path:          "/api/v1/namespaces/default/pods/somepod/exec", host:          "localhost", method:        "POST", expectAccept:  true, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
/triage accepted


Question: } } <del> if len(pods) != totalEndpointSliceAddresses { framework.Failf("Expected %d addresses, got %d", len(pods), totalEndpointSliceAddresses) </del> <ins> if len(pods) != esAddresses.Len() { framework.Failf("Expected %d addresses, got %d", len(pods), esAddresses.Len()) </ins> } }

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: lastUpdated: now, } <del> // Save the resolve, because it is still useful in case other services // are already marked dirty. THey can use it without making http request dm.setCacheEntryForService(info.service, cached) </del> <ins> // Do not save the resolve as the legacy fallback only fetches // one group version and an API Service may serve multiple // group versions. </ins> return &cached, nil case http.StatusOK:

Aggregated discovery caches when fetching from legacy discovery (). This causes apiservices who share the same service key to fetch from the cache and returns an empty document when their group-version differs from the initial apiservice that loaded the cache. See above See above No response_ 26 <detailslocal </details<details</details<details</details<details</details<details</details
/triage accepted
/sig api-machinery


Question: }, }, }, <ins> "spec.jobTemplate.spec.template.spec.restartPolicy: Unsupported value": { ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob", Namespace: metav1.NamespaceDefault, UID:       types.UID("1a2b3c"), }, Spec: batch.CronJobSpec{ Schedule:          "* * * * ?", ConcurrencyPolicy: batch.AllowConcurrent, JobTemplate: batch.JobTemplateSpec{ Spec: batch.JobSpec{ Template: api.PodTemplateSpec{ Spec: api.PodSpec{ RestartPolicy: "Invalid", DNSPolicy:     api.DNSClusterFirst, Containers:    []api.Container{{Name: "abc", Image: "image", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: api.TerminationMessageReadFile}}, }, }, }, }, }, }, </ins> } if utilfeature.DefaultFeatureGate.Enabled(features.TTLAfterFinished) { errorCases["spec.jobTemplate.spec.ttlSecondsAfterFinished:must be greater than or equal to 0"] = batch.CronJob{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deployed an cronJob with an empty and it defaulted to which is an invalid option: Leaving the key out completely has the same result What you expected to happen: The empty key to default to something valid like or . Also document what this default is, my proposal would be to have as default since some tasks can't be reran easily. But it might be nice to force as default to enforce the stateless mindset even for crons :) How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I guess not but I am open for questions :) Environment: Kubernetes version (use ): 1.19.2
My guess would be /sig api-machinery If it is better practice to wait for the triage instead of guessing the sig, let me know :+1:
/sig api-machinery
/assign
/remove-sig api-machinery /sig apps
/assign
/triage accepted


Question: } return fmt.Sprintf("%s-%d", base, id) } <ins> func checkApiServiceListQuantity(ctx context.Context, aggrclient *aggregatorclient.Clientset, label string, quantity int) func() (bool, error) { return func() (bool, error) { var err error framework.Logf("Requesting list of APIServices to confirm quantity") list, err := aggrclient.ApiregistrationV1().APIServices().List(ctx, metav1.ListOptions{LabelSelector: label}) if err != nil { return false, err } if len(list.Items) != quantity { return false, err } framework.Logf("Found %d APIService with label %q", quantity, label) return true, nil } } </ins>

[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below.

This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: } trace.Step("Computing predicates") <ins> startPredicateEvalTime := time.Now() </ins> filteredNodes, failedPredicateMap, err := findNodesThatFit(pod, g.cachedNodeInfoMap, nodes, g.predicates, g.extenders, g.predicateMetaProducer, g.equivalenceCache, g.schedulingQueue) if err != nil { return "", err

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --As of now in , we have 3 metrics, for end2endLatency, algorithmLatency and BindingLatency. The problem with algorithmLatency is it is a combination of predicate and priority latencies. So breaking down this metric into predicate and priority latencies would be really helpful for performance analysis. For example, as part of the analysis after these metrics were , we were able to find latencies when we have taints & tolerations alone. In fact, ability to drill down on every predicate and priority would be ideal but want to know other's opinions on this.
/cc
+1 please send a PR.
We also have some demonstrations of how this helps with things like highlighting taint and toleration performance and so on. We should update readme info on how to use these along side the scheduler perf stuff as we've done internally .
You need to be careful about what latency means if we are computing multiple different predicates in parallel (if we decide to split it by predicate). Splitting predicate vs priority is definitely a good idea.
Please label this as part of our 1.7 scheduling forensics milestone .
Let's remove the idea of latency entirely, it is confusing - this is just an attempt at measuring predicate and priority timescales, and there is no response latency that we care about in those small (20ms) timescales , (with the exception of maybe the latency of an etcd hit that might happen in a client if the cache is missing information I guess)
Not really - there were cases of predicates which computation could take over a second.

Yes, and even longer on large clusters with things light affinity.... However here I'm just disputing use of the term latency since latency is usually used to refer to the time between the end of a request and the beggining of a response. What we endeavor to do here is really measure round trip execution time, for predicates, in isolation. If we want to measure some kind of latency then I believe we would need to define some notion of a request/response, which has diminished value outside of something like RPC. If we truly were measuring latency (and it might be interesting to do so with tims tracing work) then we might thus look at the time between writing a new unscheduled pod to the system, and the time it takes to start running the scheduler fit functions or something... Tl;dr there are a lot of varying usages and definitions for latency, but almost all of them, with the exception of measurements of virtual function calls, refer to some unavoidable minimum cost that occurs during distributed communication, and in this case, we are simply measuring "execution time of the predicate filter" which is called from inside the same process. (Alternatively I guess it's ok to call it "fitfunctionpredicate_latency", I guess, since then we are using the fit function as the context for a response measurement).
OK, IIUC you object usage of the word 'latency', right? We use it in this context for quite some time, and as I agree that it doesn't fit formal definition, there's no point in changing it now, as it would confuse people even more.
yeah, i really think they should all be refactored to not be named as such, that is my broader point . but if we want to keep those metrics id suggest we at least update the definition in the fields to clarify that latency is defined WRT rather then to a RPC or some other typical latency calc scenario. aplogies for the bike shed, ill move this to a PR :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: return false, err } <del> updatedStorageMetrics = getControllerStorageMetrics(updatedMetrics) </del> <ins> updatedStorageMetrics = getControllerStorageMetrics(updatedMetrics, pluginName) </ins> newDetachCount, ok := updatedStorageMetrics.latencyMetrics["volume_detach"] // if detach metrics are not yet there, we need to retry

Which jobs are flaking: () () Which test(s) are flaking: Reason for failure: Timeout: Anything else we need to know: This test flaking about 10% of the time in this job. /sig storage /kind flake /priority important-soon cc
/remove-kind failing-test
/assign
I noticed one issue with the test is that it's not filtering metrics by plugin name. So it will just count the last plugin's metric. I'm not sure if the metrics collected are in any sorted order or not. If not, then when you have a suite that runs multiple plugins, then you might be pulling from the wrong metric.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
still very prevalent in sig-release dashboards
I have pushed which should hopefully fix this.
Also PR against 1.17 branch -


Question: // We expect the deployment controller to not take action here since it's configuration // is invalid, even though no replicasets exist that match it's selector. controller.syncDeployment(fmt.Sprintf("%s/%s", d.ObjectMeta.Namespace, d.ObjectMeta.Name)) <del> if len(fake.Actions()) == 0 { </del> <ins> filteredActions := filterInformerActions(fake.Actions()) if len(filteredActions) == 0 { </ins> return } <del> for _, action := range filterInformerActions(fake.Actions()) { </del> <ins> for _, action := range filteredActions { </ins> t.Logf("unexpected action: %#v", action) } t.Errorf("expected deployment controller to not take action")

https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment} Previous issues for this test:
I hit it too, direct link to build: https://k8s- PR here:
: can you try rebasing on ? That should have fixed this problem
Looks like they finally passed and retest is skipped for the submit-queue for that PR, so I'd prefer not to rebase and risk encountering another flake. If you need it as a confirmation of , let me know and I'll force push my rebase.
no worries. things are looking up in the gate. we can close this issue out tomorrow
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
This is still happening and blocking the build. please treat this as a real P0.
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
cc
opened


Question: depends on container runtime and preconfigured handler. Runtime-specific functionality is not being tested here. */ <del> ginkgo.It("should schedule a Pod requesting a RuntimeClass and initialize its Overhead [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should schedule a Pod requesting a RuntimeClass and initialize its Overhead [NodeConformance]", func() { </ins> rcName := createRuntimeClass(f, "preconfigured-handler", e2enode.PreconfiguredRuntimeClassHandler, &nodev1.Overhead{ PodFixed: v1.ResourceList{ v1.ResourceName(v1.ResourceCPU):    resource.MustParse("10m"),

Per the comment in this PR conformance tests will regress because of a newly Runtime Class GA v1 API is not covered with tests. Full comment: Full comment: /assign More links:
/sig node /triage accepted
KEP:
New to conformance tests. So the idea is to mark tests (or create new ones) that would call each of these methods at lease once? So if this test will be marked as conformance, it will cover and . Is there any additional requirements for the conformance tests? Maybe with the number of side effects of that test or a special execution environment?
Yes. You can make a single test that runs objects through CRUD operations. For the deletecollection one, I'd suggest labeling the objects under test and scoping the deletecollection to your test label to avoid breaking the cluster you're running on. See as an example
Great, will be working on it! Also just chatted with about it.
Just to get clarity on this - is this considered a blocker for Confirming bc grad criteria for it says "No release blocking feedback for API and functionality" should these tests be required to consider this stable? As a note: Code Freeze means that required tests must also be finished:
Here's possibly an easier link to track: !
This seems like something that should be tested generically, or at the very least have a table-driven test that covers all (or most) GA APIs.
! Just a quick note to show that currently the new tests are hitting all endpoints. I'd go ahead and create the Promotion PR in order to get any necessary feedback from #sig-arch
/reopen re: I disagree with the idea that a single test covering CRUD operations means RuntimeClass is now conformance tested. Generic CRUD on a resource's API is not sufficient to describe the intended behavior of the resource. That kind of test is at best verifying that api machinery is working. A vendor would able to pass this test, without implementing any of the behaviors runtimeclass enables. What does do? What kinds of behaviors are visible to an end-user? Are any of those tests in Conformance?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I'm not familiar enough with runtimeclass to know whether these would be a good start:
Thank you for reopening. I agree with this in principle. But I really would appreciate some guidance here. Scheduling tests that takes runtime handler into account are great. However they don't hit any endpoints. I asked and I'm not clear whether those needs to be covered as they are not touching an API. As for actual scheduling, one limitation is posted . Which makes some other tests impossible to include to conformance. I next week, but there may not be all the right people there as it's a Thanksgiving short week. Do you have a suggestion?
I agree that exercising specific CRI implementations is not the place of Conformance testing. Conformance testing is intended to exercise end-user visible behaviors of a Kubernetes cluster, which basically means whatever you can see and do at the Control Plane API level (ref: ) The challenge we face with Conformance testing is that we don't have an easy way to enumerate and measure coverage of behaviors. We experimented with listing behaviors in .yaml so that humans and machines could populate and approve them independent of test implementation, but ultimately abandoned that effort due to its complexity (ref: ) So we use API coverage as an easily machine-measurable/enforceable proxy. Noticing the coverage was at 0% should have been a sign that we should look at the existing test coverage of RuntimeClass and decide which tests were eligible for promotion to Conformance, and whether any new tests needed to be written. In theory a solid test plan for a KEP would include a list of behaviors that need to be tested. That's not immediately clear to me from the KEP for RuntimeClass (ref: ) Based on my read of the KEP, some valid tests might be: tests that exercise the runtimeclass admission controller creating pods with a non-existent runtimeclass creating pods with an existing runtimeclass creating pods with scheduling rules that would clash with runtimeclass scheduling rules creating pods with scheduling rules that would merge with runtimeclass scheduling rules Tests that would be disallowed by conformance requirements because they would require pre-installation/configuration of the cluster under test: running pods with a runtimeclass (unless there's some way to create a functional "dummy" runtimeclass without touching kubelets) running pods with a runtimeclass with a specific handler such as containerd, crio, etc.

and expecting them to run
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
sorry for the delay /remove-lifecycle stale
/area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: get(t, http.StatusOK) } <ins> func TestTokenPresentCancelsExecAction(t *testing.T) { a, err := newAuthenticator(newCache(), &api.ExecConfig{ Command:    "./testdata/test-plugin.sh", APIVersion: "client.authentication.k8s.io/v1alpha1", }) if err != nil { t.Fatal(err) } // UpdateTransportConfig returns error on existing TLS certificate callback, unless a bearer token is present in the // transport config, in which case it takes precedence cert := func() (*tls.Certificate, error) { return nil, nil } tc := &transport.Config{BearerToken: "token1", TLS: transport.TLSConfig{Insecure: true, GetCert: cert}} if err := a.UpdateTransportConfig(tc); err != nil { t.Error("Expected presence of bearer token in config to cancel exec action") } } </ins> func TestTLSCredentials(t *testing.T) { now := time.Now()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With the exec credentials plugin configured in my kubeconf, and a stale/expired token present in the same file issuing e.g. triggers the authentication flow as handled by the plugin - once the flow is returned to kubectl (by printing the ExecCredential object) the new token is stored to the kubeconfig but not immediately used to resolve the request, leading to: Immediately issuing again works as kubectl now seem to use the credentials (token) stored following the first request. What you expected to happen: That kubectl would read the new token as fetched by the exec credential plugin and use it already for the first command issued. How to reproduce it (as minimally and precisely as possible): the kubernetes API server with the OIDC authenticator enabled. kubectl to use an exec credential plugin printing the token received out of band. that the token was indeed written to the kubeconf file. that the first kubectl command failed. that the next identical command succeeds. Anything else we need to know?: This works as expected when there is no token previously stored in the kubeconf - the behavior is only triggered when there is a token stored but it expired, or for whatever other reason not accepted by the API server. Environment: Kubernetes version (use ): 1.14, also tried with 1.17 Cloud provider or hardware configuration: OS (e.g: ): Mac OS for kubectl client Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig auth /sig cli
could you provide more details around how is configured. For example, it is unclear to me if you are using the built in oidc plugin or a custom exec plugin. I think you are using a custom exec plugin but its also trying to write tokens to the file (which it should not do)? /triage needs-information
Thanks for looking into it . You are indeed correct in your assumption that I'm using a which stores the token in the after retrieval. I'd love to learn why this should not be done, but the problem remains also without persisting the token after retrieval.

Basically the steps to reproduce could be reduced to these: containing an expired token. same configuring an exec plugin which retrieves a valid token out of band and echoes it. Expected behavior: The valid token retrieved is used for the subsequent authentication. Possibly expected behavior: The exec plugin flow not started (or exited with a warning) due to an expired token being present in the . Current behavior seems to acknowledge the invalid token and hence tries to retrieve a new, valid one - which it then discards as soon as it arrives.
Seems we can reduce it even further by removing the token altogether from the file and instead provide it directly. So the only step needed to reproduce is essentially this: configuring an exec plugin which retrieves a valid token out of band and echoes it as part of an . Now any will trigger the exec plugin to retrieve a valid token, but will still not use it for subsequent authentication.
This one is caused by config merging with clientcmd overrides: and exec.Auth not being in this switch:
Interesting, thanks Will add it to the switch along with some tests and if everything works out well I'll submit a PR next week.
Having spent some time on this in the debugger, it seems somewhat more intricate than I first thought. Turns out the exec plugin is executed whether the presented is invalid or valid - the exec action is actually called before the first call is even dispatched. This due to the exec credential plugin possibly being needed to setup the client certificate requested for the transport. Once the exec plugin's is called it assign the authenticators function to : ..which is then called when setting up the transport, thus triggering the exec plugin regardless of a token being present or not: Since the exec plugin has no way of knowing whether a token or a client certificate will be returned by the exec action, this kind of makes sense. It however defeats letting any present to override the exec plugin, which clearly seems to be the intent: Possible options here that I can think of: (and fix) behavior of letting override the exec plugin. This looks doable by adding a check for token auth early in the 's method body: But might be there are consequences to this I'm not aware of. the behavior of letting override the exec plugin being called.

third, more complex option, would be to keep the token override but dispatch to the exec credential plugin should the response from the first call with the token come back with a 401. This was what I originally believed was the intent as my first tests always included an invalid token. While this behavior seems nice I'm more and more thinking it may violate the principle of least surprise. Either you present a token and use that for auth or you let the exec plugin fetch the credentials for you. What do you guys think?
As a colleague just made me aware of, certain tools like kubetail seem to have kubectl call the exec plugin so many times that eventually the threshold of 1000 is reached and warn logs (exec:201) start spewing - even though no certificate (and hence no rotation) should be involved. This has me more convinced that the first option is the right one, skipping the client TLS ceremonies should a token be present for authentication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "strings" "testing" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/client" )

We should support multiple objects on the command line, but, failing that, some kind of error message would be useful. /cc
Fixing, just fixed the equiv in create


Question: case "GetInodes": _, err = GetInodes(testcase.path) default: <del> t.Errorf("Case %v (%s, %v) unknown operation %s", seq, testcase.path, false, testcase.op) </del> <ins> t.Errorf("Case %v (%s, %s, %v) unknown operation %s", seq, testcase.name, testcase.path, false, testcase.op) </ins> return true } if err == nil { <del> t.Errorf("Case %v (%s, %v) %s: supports quotas but shouldn't", seq, testcase.path, false, testcase.op) </del> <ins> t.Errorf("Case %v (%s, %s, %v) %s: supports quotas but shouldn't", seq, testcase.name, testcase.path, false, testcase.op) </ins> return true } return false

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: for i, manifest := range manifests.Items { name := manifest.ID if name == "" { <del> name = fmt.Sprintf("_%d", i+1) </del> <ins> name = fmt.Sprintf("%d", i+1) </ins> } pods = append(pods, kubelet.Pod{Name: name, Manifest: manifest}) }

The hash function generates names with upper-case letters, but the validation only allows lower-case. First, we clearly need a test for this. Second, we need to fix it. Should we allow upper-case characters (which the RFC does allow for DNS labels) or should we fix the hash? Allowing upper-case seems attractive, except that the RFC also says that case doesn't matter, but should be preserved "wherever possible". So we really have three choices, I see. a) only allow lower-case b) allow upper-case and always use caseless comparisons everywhere c) allow upper-case, but normalize to lower-case since it was your hash code
Re: test - sounds like a config source -config integration test, which I'll add. Re: hash - would argue we should fix the hash to generate lowercase for now. I think, but will double check, that the collision space is still improbably low for 63 lowercase chars. I can do that shortly.
And a validate test on all sources.
Lowercase would require base32, rather than base64, no? On Aug 3, 2014 8:20 AM, "Clayton Coleman" wrote:
Yeah, although I don't believe there's a difference to the outcome between base32 and base64(lowercase) given an appropriate hash function.
Well, base32 is reversible and a modified base64is not :) On Aug 3, 2014 9:00 AM, "Clayton Coleman" wrote:
Remind me again why reversibility is important? We discussed it in another issue but I can't find the context.
I'm going to change the generated name to which even with 2^8 files per host and a thousand hosts should be well below 0.% chance of collision
Its only important if we want to use it. We don't AFAIK. On Aug 3, 2014 9:34 AM, "Clayton Coleman" wrote:
Fix makes them unique, somewhat traceable (first 15 safe chars of the name), and generates base32 output.


Question: utiltrace "k8s.io/utils/trace" ) <ins> // RulePosition holds the -I/-A flags for iptable </ins> type RulePosition string const ( <ins> // Prepend is the insert flag for iptable </ins> Prepend RulePosition = "-I" <del> Append  RulePosition = "-A" </del> <ins> // Append is the append flag for iptable Append RulePosition = "-A" </ins> ) <del> // An injectable interface for running iptables commands.  Implementations must be goroutine-safe. </del> <ins> // Interface is an injectable interface for running iptables commands.  Implementations must be goroutine-safe. </ins> type Interface interface { // EnsureChain checks if the specified chain exists and, if not, creates it.  If the chain existed, return true. EnsureChain(table Table, chain Chain) (bool, error)

Anyone itching to understand the code and try fixing things in Kubernetes, please help fixing the lint failures Step 1: Remove one or two packages (related packages) from the list in - (note: do not change generated files, the name of defaulting functions, or the name of conversion functions... packages failing linting due to issues in those files will need to be left alone) Step 2: Run and look at the logged errors Step 3: Fix the code to ensure that runs cleanly. Step 4: Run and to make sure we fix any problems with gofmt or bazel. Step 5: File a PR with the change Step 6: Go to step :) Please use the tracking spreadsheet For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
/sig architecture /sig testing
Hey, I would like to resolve this issue. Could you assign it to me ?
there are a lot of packages so there will be many people working on this. Also we cannot assign issue to folks who are not in the github org for kubernetes. please feel free to pick up a few packages like i mentioned and file a PR. Make sure you link back to this issue, so folks can see which packages you filed PR for.
OK. I see. I will firstly pick up the package called to fix.
Hi dims. Should i submit my PR before i remove one or two packages (related packages) from the list in - Or I first remove the package from list, then I fix the issue and submit a PR?
in your PR, you will have both the fixes to packages and some lines removed from /.golint_failures
When I run , I found some errors as below: I found that there is a go file in But there is still a error.
Say your GOPATH looks like this Your kubernetes directory should be () cd to and then run from that directory.
Is it necessary to obey all rules in golint? Rules in golint are set in stone. Maybe using gometalinter to ignore some specific errors is better?
I revise some erros about golint in some packages. Please review it.
which errors would you like to ignore?

if you can open a new issue with the list of errors to ignore (and proposing gometalinter), we can get some opinions there from others in the community.
Let me take the kubectl cmd part...
Can someone please review and let me know if I am doing it right?
I ll work on all the packages related to
we get 800-ish pkgs left to fix. my concern is that if we fix them package by package, then such pulls will be flooding the repo. can't we possibly make it in larger pulls? the larger the better?
+1 to slightly larger pulls with related packages. if allow too large then the reviewers will not be able to do a thorough review. we should spread the load among multiple reviewers. (if we allow a really large pull, we will end up with top level reviewers who will face the burden and the PR(s) will just stagnate)
oh good point. we could put one pkg's change into one commit, and merge all changes under one area into one pull so that it wont spread the load. wdyt? for each pull, we can have hundred-ish lines of change. but it will still be 30~50 pulls i guess? good news is that we can take our time to deal with it, at least before 1.12
sounds great! please feel free to help make it happen :) (by suggesting groups of packages that kinda go together)
I'm looking at fixing pkg/proxy, and I think I see what was talking about ignoring some rules. proxy.ProxyProvider is being suggested because golint is saying that it stutters. In that example, I think it's okay renaming it to proxy.Provider, but I could see an example where you would want to keep the stutter if the package that it's declared in uses it heavily.
sure thing, feel free to propose a PR for switching to gometalinter independently of other patches. We can work through that there.
I sent the pull request to fix pkg/volume/aws_ebs
I've fixed a few of these, but get the following error when I try to run : Are there any instructions on what I need to setup in order to resolve this?

you can find this file in
I have raised a PR to fix golint errors in pkg/api
I have fixed some pkgs under /pkg/kubelet pkg/kubelet/images pkg/kubelet/kuberuntime pkg/kubelet/leaky pkg/kubelet/pleg
I submitted a pull request for golint errors in pkg/security/ edit: I can work on picking up pkg/controller/bootstrap through pkg/controller/namespace/... (114-129)
fix golint errors in package cmd/kubeadm/app, since the codes under cmd/kubeadm/app/apis/kubeadm/v1alpha2 and cmd/kubeadm/app/apis/kubeadm/v1alpha3 are auto-generated, i keep them untouched.
I am working on some packages under pkg/kubelet, anyone interested in this plz leave messages for me :)
fix golint for pkg/util/dbus pkg/util/labels pkg/util/node pkg/util/strings pkg/util/workqueue/prometheus
PR fixes golint errors in pkg/routes,pkg/securitycontext, pkg/serviceaccount
PR fixes golint in pkg/probe
should we have a checklist as part of the issue summary saying what is in progress and what is done? Since this issue is getting long, this will make new contributors to pick up issues easier.
Great idea!, want to throw a google doc with a check-list so it's easy for participants to organize themselves? (please!) If it is the issue summary then one of us with privileges will be a bottle neck.
Created a with all the package that have on going PRs. Maybe you can add this to the summary. For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
Thanks updated the spread sheet
Raised PR to fix golint issues in: pkg/controller/clusterroleaggregation pkg/controller/cronjob pkg/controller/daemon pkg/controller/deployment pkg/controller/deployment/util pkg/controller/disruption pkg/controller/endpoint
This issue is close in accident by . could you reopen it?
/reopen
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I found a problem trying to fix golint messages for .

Golint complaints on 's function because of the snake case. When I try to fix it, fails because it is being used in a code generated file. If I clean generated files, tests pass, but the generated file is not invoking it anymore. I failed to find the generator definition using this so I could change the hook to just . Also I'm not sure if this is wanted. I also found other packages in also probably affected by this. I read somewhere about replacing golint with gometalinter, which would allow to define directives in code to omit certain rules. I would consider to have a second exclusion file for golint with these cases, to be updated if move to gometalinter is finally done. This would make this issue completion more feasible. Meanwhile, I'll move onto fixing golint messages for other packages, until this discussion gets cleared.
I also found a problem fixing . A test fails because the directory is marked as readonly (see tests @ Shall we also ignore this directory for this issue?
Is this issue still relevant because hack/verify- is giving me no errors.
the failures are still present in there are details in the first message on how to claim + solve issues :)
Sorry I misunderstood the issue
I'm working on this package:
pkg/apis/admission/v1beta1 pkg/apis/authentication/v1beta1 pkg/apis/authorization/v1beta1 pkg/apis/coordination/v1 pkg/apis/coordination/v1beta1 pkg/apis/imagepolicy/v1alpha1 pkg/apis/scheduling/v1alpha1 pkg/apis/scheduling/v1beta1 pkg/apis/settings/v1alpha1 pkg/apis/storage/v1alpha1
pkg/cloudprovider/providers/vsphere/vclib pkg/kubelet/apis/deviceplugin/v1alpha pkg/kubelet/apis/pluginregistration/v1alpha1 pkg/kubelet/apis/pluginregistration/v1beta1
:
I'm looking at the pkg/registry/*, all those which are left. I expect to give an update on what all is fixed in the next 24-48 hours.
hi there is a google sheet that is a bit outdated, maybe you can update that?
hi I seem to only have view access for the sheet. Can't edit.
I fixed the permission.

Please try it again
pkg/apis/authentication pkg/apis/coordination pkg/apis/events pkg/apis/imagepolicy pkg/apis/settings
hi I think this issue was closed prematurely by (using fixes #issue in commit)
I'll put in a PR shortly for the package
Looking at next
hi thanks! there is a spreadsheet here that tracks the progress on fixing these issues, consider adding the packages you're work on there so we can keep everything organized :+1:
Starting on now
I did indeed see the Google Sheet link (thank you and to take it a little further
test/e2e
test/e2e_node/builder
pkg/registry/admissionregistration/mutatingwebhookconfiguration/storage pkg/registry/admissionregistration/rest pkg/registry/admissionregistration/validatingwebhookconfiguration/storage pkg/registry/apps/daemonset/storage
Hi. I opened a PR - Please review.
Working on Will open a PR soon.
Starting on and should have a PR soon
I keep getting this golint verify error on and was wondering if others saw it: But when I that package to the golint failures in my PR, my reviewer said it isn't necessary? What does that mean? What did I miss? Should I use a different kind of verify? cf:
looks like the disconnect is because that problem does not show up in the version of we use in the CI. please see
Still confused: is that not the golint version I ran? it installs from vendor and uses that I think?
i installed from and i still don't see the problem, can you please check what version of golint you have?
I'm confused why I wouldn't be using the vendored version? how would that get out of sync?
/me looking
Note: I am reserving for a KubeCon Contributor Workshop demonstration. It is noted in the spreadsheet.
I think may not be ideal cause there's a lot of auto generated files there. What do you think about reserving instead for the KubeCON Contributor Workshop?
pkg/apis/scheduling pkg/apis/storage/util pkg/apis/storage/v1beta1/util pkg/apis/storage/v1/util
/priority backlog
Hi I am a new contributor here, I'd like to contribute and make changes to the following file/s. I have marked it as WIP in the Google Sheet. Can someone let me know if I have to do anything else before I start working on this PR?

Umm so there's an issue where Golint tells that underscore shouldn't be allowed. Here's a log of the go-lint warnings:
Thanks The following should not be modified, as they are designed like this for code generation.
I'd like to pick a package and do my first PR, is there a list of packages to be done? Or should I correlate with this issue thread? Thanks in advance!
please see step 1 () please scan other PRs linked to this issue as well just in case.
Will do. Thank you. On Thu, Oct 3, 2019, 7:25 AM Davanum Srinivas <:
Hi there, I'm looking for a good first issue. I'll start taking a look at if that's ok? There doesn't seem to be anyone currently tracked as looking at it in the attached spreadsheet.
I've found a few others I'm actively looking at, you won't get any argument from me. On Sat, Oct 12, 2019, 10:38 AM Pamela Kelly <:
I am working on the iptables pkg. Updated in the google docs
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have problem running in step 2 $ hack/verify- what should I do?
PTAL for my pr to this issue, thx
you can move kubernetes directory to and re-run this script.
I'd like to take Can we fix linter issues for generated files? Or we should skip packages with zz_generated?
There're 2 lint errors that affects 292/507 packages, meaning it's not possible to cleanup the entire file without solving it: default pattern of Kubectl is the usage of [cmd]Options for initial options structs and it raises: /apis/{custommetrics,externalmetrics} for example raises the underscore error, but It's probably not worth to break backward compatibility renaming these folder for lint error fix: Any advice on these?

/reopen hm, cannot reopen this issue, it was closed by mistake
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am sure this is all not fixed. Am I missing something? ./hack/verify- installing golint from vendor Congratulations! All Go source files have been linted.
you need to remove package from first.
/remove-sig testing SIG testing doesn't own code failing lints, we own the linter. :upsidedownface:
Hello, is this still a relevent issue ? I would like to work upon
I have fixed
Hi Folks, Some of the golint fixes, require us to change the exposed function's name, in simple words, "change API". I was not sure, if that is welcomed and what is the process.
Heres what Im getting after going over the following steps as mentioned by Say your GOPATH looks like this GOPATH=/Users/dims/go Your kubernetes directory should be ($) cd to and then run hack/verify- from that directory. I get the following output - installing golint Congratulations! All Go source files have been linted. Im not really sure what I am doing wrong. Any help is appreciated!
I think I understand. I misunderstood the steps around .golint_failures. The first step is to remove the pkg that we are targeting from this file and then run the My mistake was to assume that the packages in the file need to be removed after we are done cleaning the golint errors.
Can someone please review and let me know if I've done this right?
After a couple years of seeing changes spawned from this issue, I don't think this is a good use of contributors' or reviewers' time.

golint is very opinionated about: exported type and function names, leading to PRs attempting to change exported things people use downstream missing documentation on interface implementation functions, which leads to not-very-useful copy/paste of interface godoc the govet and staticcheck verify jobs are much more useful since they catch correctness issues, but I would recommend closing this issue and dropping the golint checks from CI golint doc itself discourages from using its output in an automated enforced way: from
i guess one can say gofmt is opinionated too, but something good about Golang is that it comes with an opinion and a set of standard tools. for API documentation at least we found inconsistencies on how most exported API items are documented: golint can help with that, but if there are no "pragmas" to avoid breaking changes and given we have codegen generating non-compliant names enabling golint on API packages is sort of blocked in Kubernetes. for non-API, non-library, non-public code, i think being inconsistent is mostly OK. no preference on my side. shrug
true, but code formatting has the benefits of mostly eliminating meaningless diffs in PRs due to code movement/whitespace/etc if there are specific packages (like API packages) we lack documentation on, adding checks specifically for that could make sense exactly, I don't think golint is the right tool for making sure we have API docs
golint itself doesn't have these and the go project is against this sort of comment AIUI, however golangci-lint does have them. I think there are plenty of other static analysis lints we should use instead, beyond just staticcheck and vet there is e.g.
btw, if we remove golint can we see e.g. all-caps constants or functions with underscores dodging review? perhaps we should add golangci-lint soon after:
Completely agree there are better places to spend review bandwidth. I was curious to see what the effort looked like. There have been 890 commits that touched over the past 3+ years !
/remove-help /remove-good-first-issue
Overall agree on the bandwidth. Also note that it was a good first issue with very low bar to get them in the repo. May be we need to find a similar low bar, but high value good first issue.

UT's come to mind, are there other things we curate for new contributors ?
Do we have an alternative way of enforcing that new exported code (not necessarily API fields) has at least some minimum documentation? As a reviewer, I ask contributors to add meaningful comments. But reality is that I sometimes stumble into old code that doesn't have documentation because we didn't have golint at some point. I agree that fix-golint PRs just add burden to reviewers. But, before removing the lint check, can we wait and see if it's enough to discourage people from sending more PRs like this? Additionally, we can encourage reviewers to just close PRs that only fix golint issues.
+1 for adding burden to reviewers. However, as a new contributor, I think this a very good opportunity for new contributors to study the code base and submit their first PR. In my opinion, I think this issue would be able to attract more new contributors, get familiar with the code base, and get familiar with the git workflow.
thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: At some point we can look at adding golangci-lint, there's a WIP in progress in /close
Closing this issue. <detailsIn response to : thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: MANIFEST_URL: $(yaml-quote ${MANIFEST_URL:-}) MANIFEST_URL_HEADER: $(yaml-quote ${MANIFEST_URL_HEADER:-}) NUM_NODES: $(yaml-quote ${NUM_NODES}) <del> INITIAL_ETCD_CLUSTER: $(yaml-quote ${MASTER_NAME}) </del> EOF if [ -n "${APISERVER_TEST_ARGS:-}" ]; then cat >>$file <<EOF

test grid: Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it for debs: The corresponding RPM job is also faster now:


Question: } } <ins> func getReservedCPUs(machineInfo *cadvisorapi.MachineInfo, cpus string) (cpuset.CPUSet, error) { emptyCPUSet := cpuset.NewCPUSet() if cpus == "" { return emptyCPUSet, nil } topo, err := topology.Discover(machineInfo) if err != nil { return emptyCPUSet, fmt.Errorf("Unable to discover CPU topology info: %s", err) } reservedCPUSet, err := cpuset.Parse(cpus) if err != nil { return emptyCPUSet, fmt.Errorf("Unable to parse reserved-cpus list: %s", err) } allCPUSet := topo.CPUDetails.CPUs() if !reservedCPUSet.IsSubsetOf(allCPUSet) { return emptyCPUSet, fmt.Errorf("reserved-cpus: %s is not a subset of online-cpus: %s", cpus, allCPUSet.String()) } return reservedCPUSet, nil } </ins> func run(ctx context.Context, s *options.KubeletServer, kubeDeps *kubelet.Dependencies, featureGate featuregate.FeatureGate) (err error) { // Set global feature gates based on the value on the initial KubeletServer err = utilfeature.DefaultMutableFeatureGate.SetFromMap(s.KubeletConfiguration.FeatureGates)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On a node that has a topology of more than 1 numa node, If some or all of the CPUs of a numa node are turned off (mostly for testing purposes) a valid set can cause the Kubelet service to fail on: This failure has occurred on a node that has 2 numa nodes with hyperthreading enabled. What you expected to happen: When setting a valid set, the Kubelet service should pass validations and function properly even if some CPU cores are turned off but the reserved cpus are still under a valid range. How to reproduce it (as minimally and precisely as possible): minimal cluster should have 1 master and 1 worker node. worker node should have 2 Numa nodes and hyperthreading enabled(BIOS setting) Important: The CPUs divided between the numa nodes cannot be sequential in order to reproduce the issue that are under Numa node 0 only (under ) Kubelet service off all CPUs that belong to numa node 1 () Kubelet service systemctl status kubelet.service and journal logs that will present: Anything else we need to know?: The scenario of split ranges of Numa nodes might occur only with Hyperthreading enabled. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ): (real time kernel) Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a real example where the error occurred: For the following CPU layout (after turning off numa node 1 CPUs) using : will result in: CPUs 0-2 and 40-42 are all under numa node 0 and are active.

The validation fails here: In this scenario we have 40 active CPUs (all from numa node 0) but the range check will fail on the last item in reserved CPUs: 42 (last time) = 40 (NumCores)
/cc
/sig node
This is probably not just strictly related to NUMA. The same behaviour could happen when offlining CPUs on any multi-core system, for example: disabling HT on a single numa node may end up with a very similar scenario (depends on the CPU specifics). The kubelet validation is too naive; it needs to be aware (and act accordingly to) the online CPU set. The emerging hidden assumption in kubelet is all the physical cpus are always online.
I wasn't able to recreate with all cpus on numa 1 offline, On-line CPU(s) list: 0-11,24-35 Off-line CPU(s) list: 12-23,36-47 with HT enabled, --reserved-cpus=0-3, kubelet restart fine but as mentioned by this option kubelet validation didn't consider the situation where some cpus are offline.
Ok I was able to reproduce when I used --reserved-cpus=28-35.
Just a FYI, even with the validation update, if you take certain CPUs offline, you need to delete /var/lib/kubelet/cpumanagerstate before restart kubelet. This state check is done by cpumanager, see
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale


Question: cloud.Region = region client := &testclient.Fake{} <del> controller := New(cloud, client, clusterName) </del> <ins> controller := New(cloud, client, "test-cluster2") </ins> controller.init() cloud.Calls = nil // ignore any cloud calls made in init()

Since this PR got merged, the ELB e2e test, which was consistently succeeding, is now consistently failing: Revert now that is in, and fix the tests that were relying on it. (details) Commit by arob Revert now that is in, and fix the tests that were relying on it. The file was modified The file was modified The file was modified I'm going to roll it back to confirm.


Question: framework.ExpectNoError(err, "Failed to start pod: %v", err) // sleep to make sure RequiresRepublish triggers more than 1 NodePublishVolume <del> numNodePublishVolume := 1 </del> if test.deployCSIDriverObject && csiServiceAccountTokenEnabled { time.Sleep(5 * time.Second) <del> numNodePublishVolume = 2 </del> } ginkgo.By("Deleting the previously created pod")

1 ci-kubernetes-e2e-gci-gce 1 ci-cos-containerd-e2e-cos-gce in the master blocking test grid 1 ci-kubernetes-e2e-gci-gce-network-proxy-grpc 1 ci-kubernetes-e2e-gci-gce-proto Kubernetes e2e suite.[It] [sig-storage] CSI Mock workload info CSI PodInfoOnMount Update should be passed when update from false to true 12/14 code thaw after 1.29.0 release was merged at the time and is related to PodInfoOnMount. /cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
testCase: podInfoOnMount should be passed when update from false to true kubelet log: csi-mockplugin-0-mock log: the function in the test case is not working as expected. expect log at Line 2, but actual log is at Line 3.
is called twice for the same volume sometime even though the volume is already published successfully. Not only this issue, but also other issues like I wonder if it is a potential bug or a normal behavior. cc what do you think?
good catch. I think that any test that depends on exact index of NodePublish call will be flaky, sometimes Kubelet has to retry NodePublish. It would be better to check for NodePublish parameter changes. For example, test should check there were 1 or more NodePublish calls with no pod info and then 1 or more NodePublish calls with pod info afterwards.


Question: }) } <ins> func verifyUnsupportedControllerEventEmitted(t *testing.T, dc *disruptionController) { // Verify that an UnmanagedPod event is generated found := false for e := range dc.recorder.(*record.FakeRecorder).Events { if strings.Contains(e, "managed by an unsupported controller") { found = true break } } if !found { t.Fatalf("UnmanagedPod event not generated") } } </ins> // TestMain adds klog flags to make debugging tests easier. func TestMain(m *testing.M) { klog.InitFlags(flag.CommandLine)

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { config.GinkgoConfig.SkipString = "Skipped" } gomega.RegisterFailHandler(ginkgo.Fail) // Ensure all pods are running and ready before starting tests (otherwise,

We need to be able to diagnose e2e test failures after the relevant test cluster has been shut down. Right now this almost works, but not quite. Fluentd can currently only be configured to send log files to one place, for example: Cloud Logging, or (running as a pod in the e2e test cluster, and saved to local disk on the node). With the current configuration in , logs typically go to ElasticSearch (and there is an e2e test to check that that works correctly), and tears down the e2e test cluster after each test run, so the logs are deleted. The proposed solution is to have fluentd configurable to send to multiple log collector backends, and configure it as such to both Google Cloud Logging and ElasticSearch for CI e2e test runs to facilitate easier debugging. Satnam6502@ has offered to sort this out.
Originally it was possible to send logs to both Elasticsearch and Cloud Logging but at some point this got made into a binary choice of one or the other (by or ?). I propose to undo this unless there are strong objections. : what do you think?
I didn't turn it into a binary choice. :) , Satnam Singh wrote:
Yeah, pretty sure it was Joe. I am sure it was a wide decision at the time.
Would be nice to have these at maximum verbosity because when we need them we need them all, IIRC we currently only get V(2) on a kube-up cluster
I see this is marked as a post v1.0 milestone? If we don't want this urgently then I will un-assign myself for now.
There is a valid argument that we would not block the 1.0 release for lack of these persisted test logs, and hence the v1.0 milestone was tentatively removed. There is a counter argument that we stand little chance of being able to debug the existing and still-to-be-found flaky test failures in the absence of log files, and that one or more of those bugs, if not fixed, will block the launch anyway. I'd say that if there is nothing on the v1.0 milestone that you can sensibly work on, this one is fair game.

I would find it useful, and I suppose this should be on par with writing tests (of which we have a few 1.0: )
I'm not sure we'll be able to convince folks to move this to 1.0, but feel free to work on this, as it seems very valuable.
I thought about changing our setup to allow both Elasticsearch and Cloud Logging to be targeted. I've been a little hesitant because this could lead to a breakage (esp. for GKE) and we are getting close to 1.0 ...
/cc who might have memories about a GKE breakage caused by changes in env vars used for logging.
I'd love to see this fixed soon to help debug flakes, but yes, it'd be nice to at least be cc'ed on env var changes.
Try again to have it to the high priority list, as this is making debugging the system very difficult. cc Who expressed interest.
How do we access those logs? I tried to find them for gce-scalability jenkins suite, but didn't find it.


Question: configFile     = flag.String("configfile", "/tmp/proxy_config", "Configuration file for the proxy") master         = flag.String("master", "", "The address of the Kubernetes API server (optional)") etcdServerList util.StringList <ins> bindAddress    = flag.String("bindaddress", "0.0.0.0", "The address for the proxy server to serve on (set to 0.0.0.0 or "" for all interfaces)") </ins> ) func init() {

kube-proxy listens on all ports, but should be configurable. create a flag to specify which interface to listen on. Thanks for finding this bug.
For context, some providers apparently do external IPs by having a whole interface that gets 1:1 NAT without any firewalling. We do not want service ports on those interfaces. , Brian Ketelsen wrote:


Question: if err != nil { return err } <del> added, deleted := nc.checkForNodeAddedDeleted(nodes) </del> <ins> added, deleted, newZoneRepresentatives := nc.classifyNodes(nodes) for i := range newZoneRepresentatives { nc.addPodEvictorForNewZone(newZoneRepresentatives[i]) } </ins> for i := range added { glog.V(1).Infof("NodeController observed a new Node: %#v", added[i].Name) recordNodeEvent(nc.recorder, added[i].Name, string(added[i].UID), v1.EventTypeNormal, "RegisteredNode", fmt.Sprintf("Registered Node %v in NodeController", added[i].Name)) nc.knownNodeSet[added[i].Name] = added[i] <del> // When adding new Nodes we need to check if new zone appeared, and if so add new evictor. zone := utilnode.GetZoneKey(added[i]) if _, found := nc.zoneStates[zone]; !found { nc.zoneStates[zone] = stateInitial if !nc.useTaintBasedEvictions { nc.zonePodEvictor[zone] = NewRateLimitedTimedQueue( flowcontrol.NewTokenBucketRateLimiter(nc.evictionLimiterQPS, evictionRateLimiterBurst)) } else { nc.zoneNotReadyOrUnreachableTainer[zone] = NewRateLimitedTimedQueue( flowcontrol.NewTokenBucketRateLimiter(nc.evictionLimiterQPS, evictionRateLimiterBurst)) } // Init the metric for the new zone. glog.Infof("Initializing eviction metric for zone: %v", zone) EvictionsNumber.WithLabelValues(zone).Add(0) } </del> <ins> nc.addPodEvictorForNewZone(added[i]) </ins> if nc.useTaintBasedEvictions { nc.markNodeAsHealthy(added[i]) } else {

<!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T10:51:10Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T11:29:14Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} What happened: After adding label to a node, node controller panics. What you expected to happen: Everything should still work. How to reproduce it (as minimally and precisely as possible): Run and wait for the node controller to panic: Anything else we need to know: I've checked the code and it looks like the node controller expects the label to be present during node registration and it can not be later. Even if the label can not be latter it shouldn't panic.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign let me try this :).