Question: matchByteArg(nil, actualCfg.TLSClientConfig.CAData, t) } <ins> func TestCAOverridesCAData(t *testing.T) { file, err := ioutil.TempFile("", "my.ca") if err != nil { t.Fatalf("could not create tempfile: %v", err) } defer os.Remove(file.Name()) config := createCAValidTestConfig() clientBuilder := NewNonInteractiveClientConfig(*config, "clean", &ConfigOverrides{ ClusterInfo: clientcmdapi.Cluster{ CertificateAuthority: file.Name(), }, }, nil) actualCfg, err := clientBuilder.ClientConfig() if err != nil { t.Fatalf("Unexpected error: %v", err) } matchBoolArg(false, actualCfg.Insecure, t) matchStringArg(file.Name(), actualCfg.TLSClientConfig.CAFile, t) matchByteArg(nil, actualCfg.TLSClientConfig.CAData, t) } </ins> func TestMergeContext(t *testing.T) { const namespace = "overridden-namespace"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Created cluster with , which generated its own self-signed certificate and put in . Put (API) nodes behind HTTPS ELB and for SSL termination. Run any command that has to hit the API (That's annoying because the CA that signed the cert is a public CA, and the root cert is in my OS keystore. Potentially another bug?) Download the appropriate key from (In my case, AWS ACM certificate = ) Use the certificate to attempt the same request What you expected to happen: Commandline-specified certificate overrides any other options How to reproduce it (as minimally and precisely as possible): Put any in for your host and attempt to override on cmdline. Anything else we need to know?: seems to be where the problem comes from, it's not differentiating "param came from cmdline" from "have param". If I remove the from then it works for both intended cases - either with specifying the CA cert on cmdline or via system certs. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): OSX Kernel (e.g. ):Install tools:Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Still a problem. /remove-lifecycle stale
Just ran into this as well with 1.9.0
also with 1.9.3 This is very annoying. It says "certificate-authority-data will override" but it will fail and do nothing, exiting with 0. It should at least do the override and try to execute
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Any chance of someone looking at this? Or at least freezing status, since it's evergreen until actually fixed?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale References from other tickets don't change this timer?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
any chance this can be reopened? still seems to be a problem with 1.20 of kubectl


Question: } func (rc *resourceMetricsCollector) collectNodeCPUMetrics(ch chan<- metrics.Metric, s summary.NodeStats) { <del> if s.CPU == nil { </del> <ins> if s.CPU == nil || s.CPU.UsageCoreNanoSeconds == nil { </ins> return }

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: type Interface interface { // Handle takes care of queuing and dispatching a request // characterized by the given digest.  The given `noteFn` will be <del> // invoked with the results of request classification.  If Handle // decides that the request should be executed then `execute()` // will be invoked once to execute the request; otherwise // `execute()` will not be invoked. </del> <ins> // invoked with the results of request classification.  If the // request is queued then `queueNoteFn` will be called twice, // first with `true` and then with `false`; otherwise // `queueNoteFn` will not be called at all.  If Handle decides // that the request should be executed then `execute()` will be // invoked once to execute the request; otherwise `execute()` will // not be invoked. </ins> Handle(ctx context.Context, requestDigest RequestDigest, noteFn func(fs *fctypesv1a1.FlowSchema, pl *fctypesv1a1.PriorityLevelConfiguration),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Not observed yet, but deduced from analyzing the code. If a k/apiserver/pkg/util/flowcontrol/fairqueuing.QueueSet, whose metrics.TimedObserverPair is a pair of k/apiserver/pkg/util/flowcontrol/metrics.sampleAndWaterMarkHistograms, gets zero activity for a long while and then gets some activity, then the loop at will go through a lot of iterations, and do so with a lock held. What you expected to happen: No big hiccups in any scenario. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing I noticed an incorrect fake implementation of Interface. I looked at the comment on the Handle method, and noticed that it does not say when/how the is called. What you expected to happen: I expected the parameter to be as well documented as the others. How to reproduce it (as minimally and precisely as possible): Look at and see how it does not say when/whether/with-what-parameter-values is called. Anything else we need to know?: Environment: Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Any chance we could write a test for the interface which verifies correct plumbing?
/cc
: that should be do-able


Question: func Register(containerCache kubecontainer.RuntimeCache, collectors ...prometheus.Collector) { // Register the metrics. registerMetrics.Do(func() { <ins> prometheus.MustRegister(NodeName) </ins> prometheus.MustRegister(PodWorkerLatency) prometheus.MustRegister(PodStartLatency) prometheus.MustRegister(CgroupManagerLatency)

What would you like to be : It would be useful if the kubelet exported a metric that contained its own name as a label, for instance: . This name should be consistent with the output of . Why is this needed: This would allow us to join the kubelet metrics against metrics exported by kube-state-metrics. An example of this can be seen in We've arranged in our cluster for the labels to be set to the node's name, but this is controlled by Prometheus and depends on your scrape & relabelling config. As such, in general we can't rely on this to be the case - for instance, with the prometheus-operator the label is a IP:Port combo. /cc
/sig instrumentation
This metric as it is right now doesn’t conform to the instrumentation guidelines, so yes we should changes this as part of the metrics overhaul targeted for 1.14.
you mean to add a label in kubelet metrics ?
precisely.
/assign


Question: $httpResponseMessage.Wait() if (-not $httpResponseMessage.IsCanceled) { # Check if the request was successful. <del> # </del> <ins> # </ins> # DO NOT replace with EnsureSuccessStatusCode(), it prints the # OAuth2 bearer token. if (-not $httpResponseMessage.Result.IsSuccessStatusCode) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If GCE Windows VMs are created with API scope the download GCS code will not insert the bearer token when fetching artifacts. This means that private builds of Kubernetes won't download. GCE Windows VMs should pull Kubernetes from GCS if cloud-platform scope is set on the VMs service account credentials. The cluster will not come up with Windows nodes. You can then check Cloud Logging or Serial port output for the impacted VMs and see something like, Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign me
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign jeremyje


Question: ), expectedErrs: field.ErrorList{}, }, <ins> "valid name, valid controller, invalid scope": { ingressClass: makeValidIngressClass("test123", "foo.co/bar", setParams(makeIngressClassParams(nil, "foo", "bar", nil, utilpointer.String("foo_ns"))), ), expectedErrs: field.ErrorList{ field.Required(field.NewPath("spec.parameters.scope"), ""), }, }, </ins> "namespace not set when scope is Namespace": { ingressClass: makeValidIngressClass("test123", "foo.co/bar", setParams(makeIngressClassParams(nil, "foo", "bar", utilpointer.String("Namespace"), nil)),

This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It looks like you are correct. Please feel free to submit a pull request to fix it. For information on contribute, please refer to:
/sig network /sig auth
/assign
when reporting issues please give more context to explain it, in this case it is required so it never can be nil in this comparison
Who wants to fix this? This whole block is suspect - it checks for nil later, too.
I'm working on it in !


Question: } // Finish housekeeping. <ins> // Clear stale conntrack entries for UDP Services, this has to be done AFTER the iptables rules are programmed. </ins> // TODO: these could be made more consistent. <del> klog.V(4).InfoS("Deleting stale services", "ips", staleServices.UnsortedList()) for _, svcIP := range staleServices.UnsortedList() { </del> <ins> klog.V(4).InfoS("Deleting conntrack stale entries for Services", "ips", conntrackCleanupServiceIPs.UnsortedList()) for _, svcIP := range conntrackCleanupServiceIPs.UnsortedList() { </ins> if err := conntrack.ClearEntriesForIP(proxier.exec, svcIP, v1.ProtocolUDP); err != nil { klog.ErrorS(err, "Failed to delete stale service connections", "ip", svcIP) } } <ins> klog.V(4).InfoS("Deleting conntrack stale entries for Services", "nodeports", conntrackCleanupServiceNodePorts.UnsortedList()) for _, nodePort := range conntrackCleanupServiceNodePorts.UnsortedList() { err := conntrack.ClearEntriesForPort(proxier.exec, nodePort, isIPv6, v1.ProtocolUDP) if err != nil { klog.ErrorS(err, "Failed to clear udp conntrack", "port", nodePort) } } </ins> klog.V(4).InfoS("Deleting stale endpoint connections", "endpoints", endpointUpdateResult.StaleEndpoints) proxier.deleteEndpointConnections(endpointUpdateResult.StaleEndpoints) }

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: proxier.servicesSynced = true } <ins> func makeEndpointSliceMap(proxier *Proxier, allEpSlices ...*discovery.EndpointSlice) { for i := range allEpSlices { proxier.OnEndpointSliceAdd(allEpSlices[i]) } proxier.mu.Lock() defer proxier.mu.Unlock() proxier.endpointSlicesSynced = true } </ins> func makeTestService(namespace, name string, svcFunc func(*v1.Service)) *v1.Service { svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{

Entry in ipvs real sever remains at weight=0 after kube-proxy restarts. (During 1.21 -1.22 kubernetes cluster upgrade.) Related to As the related issue is considered closed, I will report this issue again as the problem still persists in newer version. Might because of some corner cases. Real server weight should be 1 when endpoints is ready. Use kube-proxy in ipvs mode Deploy a service resource and its backend pods Make real sever not ready (or may need node not ready) so as to trigger graceful delete in kube-proxy Before kube-proxy deletes the entry from ipvs, restart kube-proxy Kube-proxy becomes ready and leave the weight=0 entry as it is Service is no longer accessible through cluster ip When accessed, dmesg shows kernel messge error ofDeletion of pod will trigger kube-proxy to remove weight=0 entry from ipvs so that after the entry is recreated service will become accessible If service is 1. LoadBalancer, and, 2. .spec.externalTrafficPolicy: Local, and, 3, service is accessed from loadbalancer IP, the entry in ipvs for loadbalancer IP will only have endpoints in the same node. If backends is DaemonSet, the only one endpoints will have weight=0 thus causing connection failure After kube-proxy restarts, endpoints will be processed in the following lines. Only when the entry is in the gracefuldeleteManager.InTerminationList will be deleted. Which is not the case if kube-proxy restarts before it deletes the ipvs entry. An existing entry in the ipvs is sufficient for endpoint to be considered healthy. Weight is not considered. kube-proxy should treat ipvs entry properly by the either the following two ways. kube-proxy should initiate InTerminationList and put ipvs entry with weight=0 into the termination list kube-proxy should also look for weight in the ipvs entry instead of only UniqueRSName <details</details<detailson-premises </details<details</details<details</details<details</details<details</details
/sig network
Graceful delete of ipvs is not only triggered when pod is in terminating phase. Graceful delete seems to be triggered when endpointslice status is ready=false. This will happen when node become not ready.

However, after the node become ready again, pods on the node will only be recreated if the duration of NotReady is longer than certain time. Otherwise the pod will just become ready again. If the node will not recreate the pods && kube-proxy set the pods to the graceful delete queue && kube-proxy restarts before deletion and all cache is cleared, kube-proxy will keep ipvs entry with weight=0 even if the real server is ready. Please correct me if I misunderstand something.
/area ipvs /area kube-proxy
Seems like a legit bug to me. I can't triage until wednesday though.
I can't recreate this when I try to follow thre instructions with pod not-ready rather than node not ready. When the pod is made ready again is not stated, so I tried; a connection to a pod it not-ready. The weight is set to 0, the connection prevails -start kube-proxy. The weight is still 0, connection still working the pod ready weight is set to 1 after a while, as it should
In the instruction is different, the pod is made ready while is away. That instruction shows the fault. The reproduce instruction in this issue is incorrect.
I triage this issue as a bug, but it's really a duplicate of which was never fixed (it seems). Don't know if the proper action is to close this issue and re-open . /triage accepted
Please instruct if this issue shall be closed as duplicate and re-opened PTAL at
Thank you for your triage Monitoring IPVS weights is costly, but needed while this problem persists. Any news? Adding the following lines should fix the problem but will also add loads.
That does seem costly to do on every endpoint sync. Is there an alternative solution we can explore where we assess all endpoints once when kube-proxy initially starts up?
This is a startup-only problem. It should be solved with a one-time action.
How about modifying to be ? We could then check and update the real server weights only when is true. Only and will be modified to call , all other users will call the function with false. Since these two functions, are only called during startup, we won't be doing this monitoring every endpoint sync.
cc:


Question: timeoutPermit       bool waitAndRejectPermit bool waitAndAllowPermit  bool <ins> allowPermit         bool </ins> fh                  framework.FrameworkHandle }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!


Question: "context" "errors" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime"

The 's was copied from some other, broken types. A call to is missing and therefore, for example, finalizers can be changed during a status update. That is not supposed to be possible. Object meta changes need to be removed during status updates. n/a No response Since 1.26 till at least 1.29. n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign


Question: "k8s.io/apiserver/pkg/endpoints/metrics" apirequest "k8s.io/apiserver/pkg/endpoints/request" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" <del> "k8s.io/apiserver/pkg/server/httplog" </del> "github.com/golang/glog" )

Example call stack from : We should fix this and add an integration test that sets the limit low and exercises apiserver when it is getting too many requests.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Should be fixed by the above PR. Closing it.


Question: features featuregate.FeatureGate } <ins> var _ = authorizer.Authorizer(&NodeAuthorizer{}) var _ = authorizer.RuleResolver(&NodeAuthorizer{}) </ins> // NewAuthorizer returns a new node authorizer <del> func NewAuthorizer(graph *Graph, identifier nodeidentifier.NodeIdentifier, rules []rbacv1.PolicyRule) authorizer.Authorizer { </del> <ins> func NewAuthorizer(graph *Graph, identifier nodeidentifier.NodeIdentifier, rules []rbacv1.PolicyRule) *NodeAuthorizer { </ins> return &NodeAuthorizer{ graph:      graph, identifier: identifier,

What happened: I checked if kubelet () has permissions for with SelfsubjectAccessReviews and I saw that it has. But when I checked its permissions with SelfsubjectRulesReviews it didn't show it. What you expected to happen: I expected that the API SelfsubjectRulesReviews will show that has permssions for . How to reproduce it (as minimally and precisely as possible): You can check it on the master or the worker. Test SelfsubjectAccessReviews: You will get answer that it has permissions: Test SelfsubjectRulesReviews: It won't show the permissions. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm /area/api /kind/bug /sig api-machinery
/sig api-machinery
This is a more clean display that show the bug: I think the problem is related to the fact that the file contain the kubelet certificate which is part of the group :
/sig auth
What is the SelfSubjectRulesReview response from the server ()? Note that not all authorizers support enumerating permissions, and the API object status includes an attribute which explicitly indicates the result is incomplete. See
It has Details:
I noticed that it also doesn't show that it has permissions to view and . It probably related to the Node authorizer: Because I assume that these rules doesn't appear in RBAC.
The node authorizer does not enumerate permissions, so it does not contribute to SelfSubjectRulesReview results
fixes setting correctly when the API is requested with a kubelet's credentials


Question: { Name:    "client-container", Image:   "gcr.io/google_containers/mounttest:0.6", <del> Command: []string{"/mt", "--break_on_expected_content=false", "--retry_time=10", "--file_content_in_loop=" + filePath}, </del> <ins> Command: []string{"/mt", "--break_on_expected_content=false", "--retry_time=120", "--file_content_in_loop=" + filePath}, </ins> VolumeMounts: []api.VolumeMount{ { Name:      "podinfo",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig network
thought you might have some insight here?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
will try to check later, but would be great if someone can verify this is not a duplicate of
looks like it might be, any idea why this would have just started now on the 1.19 branch?
seems the job started to run networking tests on friday :thinking:

In parallel suite:
Can you recreate this? It could be a race or a buffer not flushing.
I'm having a look
Anoter example of the failure: - awesome thanks!
unluckily I cannot seet this page. Thanks anyway
to delegate? (If you don't think it's a node bug, maybe can find an owner?)
Here's our latest conformance log from master with the labels/annotations both failing (mesos/docker cluster): Downward API volume should update labels on modification [Conformance] Downward API volume should update annotations on modification [Conformance]
These failures can be reproduced on master with a single command:
thanks for the logs. I'm investigating the issue using (never worked with so I'm not sure if it's the same or not) Running only one instance of like this --host="127.0.0.1:8080" --provider="local" --ginkgo.v=true --"*on.modification" --kubeconfig="$" basically your command (I guess) I never had a failure. Running the same command in a loop it's different and I managed to reproduce the issue, most of the time the timeout looks to aggressive 15s (while in most of the tests the timeout is something like 5 minutes). since it's seems that this is blocking for several person I'd like to raise the timout (at least to 30seconds). I'm not saying that this is the fix (the old joke about ) but at least it should reduce the problem on busy system.
It looks to me that everytime the fails happens I ran into something like: the same is valid for logs. I tried to in the code (maybe I'm blind) but I couldn't find... Any ideas?
- those lines are coming from this metric: the "kubelet_" at the beginning is "subsystem" in prometheus metrics, that's why you probably didn'g find it
Actually it's printed here:
thanks thanks
We should verify if increasing the timeout: fixed this issue.
Still fails on the mesos/docker cluster on master:
I have been informed that the mesos executor wasn't actually handling annotation/label updates. So these tests were failing legitimately on the mesos/docker cluster.

This will hopefully be fixed by
don't know if makes sense for you but kubelet broke this and we add to fix
Another failure:
:( Sorry I didn't see this... Ok. Having a look again (but today I'm pretty busy)
is not enough. We need also to increase timeout in image which generates the logs sorry for this I didn't realize it when we talked about this thanks for the file!
submitted to increase timout in running image
- great - thanks for looking into it!
I'm reopening it, so that we can monitor if this really fixed the problem. Will close it if we don't observe more errors for a couple of days.
There weren't any failures of this test within last 3 days - closing! Thanks
Saw this flake again in a GKE run:
same test but another issue. we got this: . While the previous issue was a too short timeout (container exited too early for slow envs). would you like to open another issue for this?
Working on this to propose a fix
submitted please have a look
I saw you xref this issue from . I cannot see the relations (and I could be wrong). Mind explain? Let me know if I need to Thanks
- the test you are trying to fix failed in the presubmit of that PR: I didn't review but it's not merged yet, so we don't know if that helps (hopefully yes :))
I think will help on these kind of issues (retrying on the update). thanks for the explanation.
I encountered a failure of this test for a different reason, I can create a separate issue if you think it's unrelated: link:
In the current implementation is performed without retry. implement a retry mechanism similar to
thanks. FYI, Actually we can set the Pod.ResourceVersion="", that will force an update, it would be cleaner than using a Poll.
I'm going to update the PR with what you're suggesting. Thanks


Question: evictionMonitoringPeriod = time.Second * 10 // The path in containers' filesystems where the hosts file is mounted. <del> etcHostsPath = "/etc/hosts" </del> <ins> linuxEtcHostsPath   = "/etc/hosts" windowsEtcHostsPath = "C:WindowsSystem32driversetchosts" </ins> // Capacity of the channel for receiving pod lifecycle events. This number // is a bit arbitrary and may be adjusted in the future.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: When deploying Kubernetes pods, the Linux containers will contain the following entries in the file: # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.240.0.67 dns-querier-1.dns-test-service.e2e-tests-dns- dns-querier-1 However, these are not set on Windows containers; the container's file does not contain any entry. This can be attributed to the fact that individual files cannot be mounted on Windows containers the same way they are on Linux containers. It can be fixed with a change to the kubelet once Windows Server 2019 has shipped, and once CRI-ContainerD is enabled on Windows. We cannot fix it for Windows Server 1803 or earlier with dockershim. What you expected to happen: The file should be managed and populated by Kubelet. How to reproduce it (as minimally and precisely as possible): Spawn a Windows pod, and check its container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cmd /S /C type C:WindowsSystem32driversetchosts Compare the result with a Linux pod's container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cat /etc/hosts Anything else we need to know?: Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: N/A OS (e.g.

from /etc/os-release): Windows Server 1803 (10.0..48) Install tools: acs-engine v0.24.0 Others:
This requires single-file mapping, which won't work in Windows Server version 1803. It should be possible with Windows Server 2019 along with ContainerD
Will it work with Docker on Windows Server 2019?
Not in 18.09 or earlier. I'm not sure what release it will be in because it will require migrating to containerd.
should this issue be tracked for 1.13 in addition to If so please apply the appropriate priority and milestone labels please.
This one needs to be in v1.14. We need to get cri-containerd working first to fix this.
Thanks /milestone v1.14
By default, hosts file are not being used in Windows, it is all registry based. What features is this blocking? CC
The hosts file entries are still taken into account on Windows when it has to resolve a DNS name. This is blocking DNS conformance tests, like.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
Hello! 14 code freeze is coming in about 3 days, is this intended to be implemented in the following week?
No. It is meant to be fixed in the next release, when the Containerd support will be introduced.
/milestone v1.15
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
This is dependent of the ContainerD implementation. If that hasn't been implemented yet, this can't either.
Is there any update is the containerd implementation finished? Are there any relevant issues or PRs that we can use for tracking?
sounds like a no-go for 1.15, let's just move it out as no one is working on it as well. /milestone v1.16
Hello!

I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
/milestone v1.17 We won't be able to do this until containerd is ready. It might work in 1.17
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
We have a PR for it, as you can see above: It;s a pretty small PR too.
The v1.17 code freeze is this Thursday, end of day Pacific time. Will this issue be closed by then?
/milestone v1.18 (code freeze v1.17)
This is going to need a Windows fix, not a ContainerD fix. We won't have that before code freeze on 5 March, so I'm pushing this out to 1.19. /milestone v1.19
This is being tracked with MS internal work item The Windows team thinks an OS fix for Windows Server 2019 will be released with the June patches. I will post updates here as I get them.
Hello from the bug triage team! This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
can you comment on when we can expect a Windows fix for Windows Server 2019 that addresses the single file mount issues in containers when a target file already exists in the container?

we are waiting on a Windows OS fix from Microsoft to proceed here. If that fix comes in time for us to validate this functionality in 1.19 we would like it to be included in 1.19.
Hello! As this issue seems to depend on PR which has been pushed to v1.20, should this issue also be pushed to that milestone?
/milestone 1.20 The OS fix is now expect to be available as part of August Windows patches and then changes to the kubelet will need to follow. /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone.

<detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
yes it is expected be part of August Windows release
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
- Yes, we'd like to make it into 1.20. We were waiting on some bugfixes in the Windows OS and those bugfixes are now available (as of Oct 20 2020).
/remove-lifecycle frozen
/priority important-soon
I believe the work involved here is to update to enable this fucntionality for WIndows if container-runtime == remote (still disable this for DockerShim) Update/Add test coverage Update documentation - possible this section ?
/milestone v1.21 Moving to 1.21 since the required OS fixes were not available in time to implement this functionality in v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello! I am from the Bug Triage team! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
Can this be reopened? From a related problem I have experienced using HostAlias for Windows pod deployments, this issue does not seem to be resolved. Instead, it looks as if it was closed by accident because the conversation went cold.
Hello. What Kubernetes version are you running? Also, on the Windows nodes, do you use docker, or containerd?

This is only supported on Windows nodes with containerd, since it supports single file mapping, which is required in order to mount single files into Windows containers.
We have experimented with both 1.22.11 and a 1.23 release and yea, the container runtime is containerd. Any other ideas to check up on?


Question: // initialization. This *noise* is obvious when N is small. We // deliberately set higher resource usage limits to account for the // noise. <ins> // // We set all resource limits generously because this test is mainly // used to catch resource leaks in the soak cluster. For tracking // kubelet/runtime resource usage, please see the node e2e benchmark // dashboard. http://node-perf-dash.k8s.io/ // // TODO(#36621): Deprecate this test once we have a node e2e soak // cluster. </ins> rTests := []resourceTest{ { podsPerNode: 0, cpuLimits: framework.ContainersCPUSummary{ <del> stats.SystemContainerKubelet: {0.50: 0.06, 0.95: 0.08}, stats.SystemContainerRuntime: {0.50: 0.05, 0.95: 0.06}, </del> <ins> stats.SystemContainerKubelet: {0.50: 0.10, 0.95: 0.20}, stats.SystemContainerRuntime: {0.50: 0.10, 0.95: 0.20}, </ins> }, <del> // We set the memory limits generously because the distribution // of the addon pods affect the memory usage on each node. </del> memLimits: framework.ResourceUsagePerContainer{ stats.SystemContainerKubelet: &framework.ContainerResourceUsage{MemoryRSSInBytes: 70 * 1024 * 1024}, // The detail can be found at https://github.com/kubernetes/kubernetes/issues/28384#issuecomment-244158892

https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite} Previous issues for this test:
Looks like a duplicate of . Assigning it to for further triaging.
failing in https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- https://k8s-
dup
sorry, not a dup
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: // have a place they can schedule. cfg2 := config.NewPodConfig(config.PodConfigNotificationSnapshotAndUpdates) config.NewSourceEtcd(config.EtcdKeyForHost(machineList[1]), etcdClient, cfg2.Channel("etcd")) <del> otherKubelet := kubelet.NewIntegrationTestKubelet(machineList[1], &fakeDocker2) </del> <ins> otherKubelet := kubelet.NewIntegrationTestKubelet(machineList[1], testRootDir, &fakeDocker2) </ins> go util.Forever(func() { otherKubelet.Run(cfg2.Updates()) }, 0) go util.Forever(func() { kubelet.ListenAndServeKubeletServer(otherKubelet, cfg2.Channel("http"), "localhost", 10251)

E1003 10:53:11. ] Could not read directory: , (open : no such file or directory) This happens many times


Question: BackendIPConfigurations: &[]network.InterfaceIPConfiguration{ { Name: to.StringPtr("ip-1"), <del> ID:   to.StringPtr("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/0/networkInterfaces/nic"), </del> <ins> ID:   to.StringPtr("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss-1/virtualMachines/0/networkInterfaces/nic"), </ins> }, }, },

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestEnsureBackendPoolDeletedConcurrently Testgrid link: Reason for failure: /sig cloud-provider
cc
/assign /area provider/azure


Question: // This method will not be called from the node that is requesting this ID. i.e. metadata service // and other local methods cannot be used here func (az *Cloud) InstanceTypeByProviderID(providerID string) (string, error) { <del> return "", errors.New("unimplemented") </del> <ins> name, err := splitProviderID(providerID) if err != nil { return "", err } return az.InstanceID(name) </ins> } // InstanceType returns the type of the specified instance.

This is required to make Cloud Controller Manager operational on the various clouds that Kubernetes runs on. We have already PRs for the following clouds: GCE - AWS - Azure - OpenStack - The following clouds still need to be addresses: CloudStack Mesos OVirt Photon Rackspace vSphere If anyone is interested in contributing to this effort, please ping me directly (on kubernetes slack or through email)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-lifecycle
cc:
I will take care of the vSphere and Photon.
I will take care of the Rackspace.
Merging for OpenStack
Reposting I am trying to use my custom out-of-tree cloud provider with Kubernetes. But I noticed that kubelet calls apis like with empty for cloud provider. Digging in the code for 1.7 branch, I see that is null for cloud-providers. As a result returns empty string. As a result, ByID() methods don't work for cloud providers since is not set.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: endif build: <del> # Copy the content in this dir to the temp dir, # without copying the subdirectories. find ./ -maxdepth 1 -type f | xargs -I {} cp {} $(TEMP_DIR) </del> <ins> # Explicitly copy files to the temp directory $(BIN_INSTALL) migrate-if-needed.sh $(TEMP_DIR) install Dockerfile $(TEMP_DIR) </ins> # Compile migrate <del> docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes -v $(TEMP_DIR):/build -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION)  /bin/bash -c "CGO_ENABLED=0 go build -o /build/migrate k8s.io/kubernetes/cluster/images/etcd/migrate" </del> <ins> migrate_tmp_dir=$(shell mktemp -d);  docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes$(DOCKER_VOL_OPTS) -v $${migrate_tmp_dir}:/build$(DOCKER_VOL_OPTS) -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION)  /bin/bash -c "CGO_ENABLED=0 go build -o /build/migrate k8s.io/kubernetes/cluster/images/etcd/migrate";  $(BIN_INSTALL) $${migrate_tmp_dir}/migrate $(TEMP_DIR) </ins> ifeq ($(ARCH),amd64)

Part of What happened: Attempt to run as a non-root user Get What you expected to happen: It runs How to reproduce it (as minimally and precisely as possible): /sig release /sig cluster-lifecycle /area security And as part of kubeadm v1.16 milestone: /milestone v1.16 /priority critical-urgent
/assign randomvariable /lifecycle active
/priority important-soon


Question: assert.Equal(t, c.expected, loadbalancerName, c.description) } } <ins> func TestGetLoadBalancingRuleName(t *testing.T) { az := getTestCloud() az.PrimaryAvailabilitySetName = "primary" svc := &v1.Service{ ObjectMeta: meta.ObjectMeta{ Annotations: map[string]string{}, UID:         "257b9655-5137-4ad2-b091-ef3f07043ad3", }, } cases := []struct { description   string subnetName    string isInternal    bool useStandardLB bool protocol      v1.Protocol port          int32 expected      string }{ { description:   "internal lb should have subnet name on the rule name", subnetName:    "shortsubnet", isInternal:    true, useStandardLB: true, protocol:      v1.ProtocolTCP, port:          9000, expected:      "a257b965551374ad2b091ef3f07043ad-shortsubnet-TCP-9000", }, { description:   "internal standard lb should have subnet name on the rule name but truncated to 80 characters", subnetName:    "averylonnnngggnnnnnnnnnnnnnnnnnnnnnngggggggggggggggggggggggggggggggggggggsubet", isInternal:    true, useStandardLB: true, protocol:      v1.ProtocolTCP, port:          9000, expected:      "a257b965551374ad2b091ef3f07043ad-averylonnnngggnnnnnnnnnnnnnnnnnnnnnngg-TCP-9000", }, { description:   "internal basic lb should have subnet name on the rule name but truncated to 80 characters", subnetName:    "averylonnnngggnnnnnnnnnnnnnnnnnnnnnngggggggggggggggggggggggggggggggggggggsubet", isInternal:    true, useStandardLB: false, protocol:      v1.ProtocolTCP, port:          9000, expected:      "a257b965551374ad2b091ef3f07043ad-averylonnnngggnnnnnnnnnnnnnnnnnnnnnngg-TCP-9000", }, { description:   "external standard lb should not have subnet name on the rule name", subnetName:    "shortsubnet", isInternal:    false, useStandardLB: true, protocol:      v1.ProtocolTCP, port:          9000, expected:      "a257b965551374ad2b091ef3f07043ad-TCP-9000", }, { description:   "external basic lb should not have subnet name on the rule name", subnetName:    "shortsubnet", isInternal:    false, useStandardLB: false, protocol:      v1.ProtocolTCP, port:          9000, expected:      "a257b965551374ad2b091ef3f07043ad-TCP-9000", }, } for _, c := range cases { if c.useStandardLB { az.Config.LoadBalancerSku = loadBalancerSkuStandard } else { az.Config.LoadBalancerSku = loadBalancerSkuBasic } svc.Annotations[ServiceAnnotationLoadBalancerInternalSubnet] = c.subnetName svc.Annotations[ServiceAnnotationLoadBalancerInternal] = strconv.FormatBool(c.isInternal) loadbalancerRuleName := az.getLoadBalancerRuleName(svc, c.protocol, c.port) assert.Equal(t, c.expected, loadbalancerRuleName, c.description) } } func TestGetFrontendIPConfigName(t *testing.T) { az := getTestCloud() az.PrimaryAvailabilitySetName = "primary" svc := &v1.Service{ ObjectMeta: meta.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationLoadBalancerInternalSubnet: "subnet", ServiceAnnotationLoadBalancerInternal:       "true", }, UID: "257b9655-5137-4ad2-b091-ef3f07043ad3", }, } cases := []struct { description   string subnetName    string isInternal    bool useStandardLB bool expected      string }{ { description:   "internal lb should have subnet name on the frontend ip configuration name", subnetName:    "shortsubnet", isInternal:    true, useStandardLB: true, expected:      "a257b965551374ad2b091ef3f07043ad-shortsubnet", }, { description:   "internal standard lb should have subnet name on the frontend ip configuration name but truncated to 80 characters", subnetName:    "averylonnnngggnnnnnnnnnnnnnnnnnnnnnngggggggggggggggggggggggggggggggggggggsubet", isInternal:    true, useStandardLB: true, expected:      "a257b965551374ad2b091ef3f07043ad-averylonnnngggnnnnnnnnnnnnnnnnnnnnnnggggggggggg", }, { description:   "internal basic lb should have subnet name on the frontend ip configuration name but truncated to 80 characters", subnetName:    "averylonnnngggnnnnnnnnnnnnnnnnnnnnnngggggggggggggggggggggggggggggggggggggsubet", isInternal:    true, useStandardLB: false, expected:      "a257b965551374ad2b091ef3f07043ad-averylonnnngggnnnnnnnnnnnnnnnnnnnnnnggggggggggg", }, { description:   "external standard lb should not have subnet name on the frontend ip configuration name", subnetName:    "shortsubnet", isInternal:    false, useStandardLB: true, expected:      "a257b965551374ad2b091ef3f07043ad", }, { description:   "external basic lb should not have subnet name on the frontend ip configuration name", subnetName:    "shortsubnet", isInternal:    false, useStandardLB: false, expected:      "a257b965551374ad2b091ef3f07043ad", }, } for _, c := range cases { if c.useStandardLB { az.Config.LoadBalancerSku = loadBalancerSkuStandard } else { az.Config.LoadBalancerSku = loadBalancerSkuBasic } svc.Annotations[ServiceAnnotationLoadBalancerInternalSubnet] = c.subnetName svc.Annotations[ServiceAnnotationLoadBalancerInternal] = strconv.FormatBool(c.isInternal) ipconfigName := az.getFrontendIPConfigName(svc) assert.Equal(t, c.expected, ipconfigName, c.description) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: <ins> # Availability This document collects advice on reasoning about and provisioning for high-availability when using Kubernetes clusters. ## Failure modes This is an incomplete list of things that could go wrong, and how to deal with them. Root causes: - VM(s) shutdown - network partition within cluster, or between cluster and users. - crashes in Kubernetes software - data loss or unavailability of persistent storage (e.g. GCE PD or AWS EBS volume). - operator error misconfigures kubernetes software or application software. Specific scenarios: - Apiserver VM shutdown or apiserver crashing - Results - unable to stop, update, or start new pods, services, replication controller - existing pods and services should continue to work normally, unless they depend on the Kubernetes API - Apiserver backing storage lost - Results - apiserver should fail to come up. - kubelets will not be able to reach it but will continute to run the same pods and provide the same service proxying. - manual recovery or recreation of apiserver state necessary before apiserver is restarted. - Supporting services (node controller, replication controller manager, scheduler, etc) VM shutdown or crashes - currently those are colocated with the apiserver, and their unavailability has similar consequences as apiserver - in future, these will be replicated as well and may not be co-located - they do not have own persistent state - Node (thing that runs kubelet and kube-proxy and pods) shutdown - Results - pods on that Node stop running - Kubelet software fault - Results - crashing kubelet cannot start new pods on the node - kubelet might delete the pods or not - node marked unhealthy - replication controllers start new pods elsewhere - Cluster operator error - Results: - loss of pods, services, etc - lost of apiserver backing store - users unable to read API - etc Mitigations: - Action: Use IaaS providers automatic VM restarting feature for IaaS VMs. - Mitigates: Apiserver VM shutdown or apiserver crashing - Mitigates: Supporting services VM shutdown or crashes - Action use IaaS providers reliable storage (e.g GCE PD or AWS EBS volume) for VMs with apiserver+etcd. - Mitigates: Apiserver backing storage lost - Action: Use Replicated APIserver feature (when complete: feature is planned but not implemented) - Mitigates: Apiserver VM shutdown or apiserver crashing - Will tolerate one or more similtaneous apiserver failures. - Mitigates: Apiserver backing storage lost - Each apiserver has independent storage.  Etcd will recover from loss of one member.  Risk of total data loss greatly reduced. - Action: Snapshot apiserver PDs/EBS-volumes periodically - Mitigates: Apiserver backing storage lost - Mitigates: Some cases of operator error - Mitigates: Some cases of kubernetes software fault - Action: use replication controller and services in front of pods - Mitigates: Node shutdown - Mitigates: Kubelet software fault - Action: applications (containers) designed to tolerate unexpected restarts - Mitigates: Node shutdown - Mitigates: Kubelet software fault - Action: Multiple independent clusters (and avoid making risky changes to all clusters at once) - Mitigates: Everything listed above. ## Chosing Multiple Kubernetes Clusters You may want to set up multiple kubernetes clusters, both to have clusters in different regions to be nearer to your users; and to tolerate failures and/or invasive maintenance. ### Scope of a single cluster On IaaS providers such as Google Compute Engine or Amazon Web Services, a VM exists in a [zone](https://cloud.google.com/compute/docs/zones) or [availability zone](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). We suggest that all the VMs in a Kubernetes cluster should be in the same availability zone, because: - compared to having a single global Kubernetes cluster, there are fewer single-points of failure - compared to a cluster that spans availability zones, it is easier to reason about the availability properties of a single-zone cluster. - when the Kubernetes developers are designing the system (e.g. making assumptions about latency, bandwidth, or correlated failures) they are assuming all the machines are in a single data center, or otherwise closely connected. It is okay to have multiple clusters per availability zone, though on balance we think fewer is better. Reasons to prefer fewer clusters are: - improved bin packing of Pods in some cases with more nodes in one cluster. - reduced operational overhead (though the advantage is diminished as ops tooling and processes matures). - reduced costs for per-cluster fixed resource costs, e.g. apiserver VMs (but small as a percentage of overall cluster cost for medium to large clusters). Reasons to have multiple clusters include: - strict security policies requiring isolation of one class of work from another (but, see Partitioning Clusters below). - test clusters to canary new Kubernetes releases or other cluster software. ### Selecting the right number of clusters The selection of the number of kubernetes clusters may be a relatively static choice, only revisted occasionally. By contrast, the number of nodes in a cluster and the number of pods in a service may be change frequently according to load and growth. To pick the number of clusters, first, decide which regions you need to be in to have adequete latency to all your end users, for services that will run on Kubernetes (if you use a Content Distribution Network, the latency requirements for the CDN-hosted content need not be considered).  Legal issues might influence this as well. For example, a company with a global customer base might decide to have clusters in US, EU, AP, and SA regions. Call the number of regions to be in `R`. Second, decide how many clusters should be able to be unavailable at the same time, while still being available.  Call the number that can be unavailable `U`.  If you are not sure, then 1 is a fine choice. If it is allowable for load-balancing to direct traffic to any region in the event of a cluster failure, then then you need `R + U` clusters.  If it is not (e.g you want to ensure low latency for all users in the event of a cluster failure), then you need to have `R * U` clusters (`U` in each of `R` regions).  In any case, try to put each cluster in a different zone. Finally, if any of your clusters would need more than the maximum recommended number of nodes for a Kubernetes cluster, then you may need even more clusters.  Our roadmap ( https://github.com/GoogleCloudPlatform/kubernetes/blob/24e59de06e4da61f5dafd4cd84c9340a2c0d112f/docs/roadmap.md) calls for maximum 100 node clusters at v1.0 and maximum 1000 node clusters in the middle of 2015. ## Working with multiple clusters When you have multiple clusters, you would typically create services with the same config in each cluster and put each of those service instances behind a load balancer (AWS Elastic Load Balancer, GCE Forwarding Rule or HTTP Load Balancer), so that failures of a single cluster are not visible to end users. </ins>

<!-- STOP -- PLEASE READ! GitHub is not the right place for support requests. If you're looking for help, check and the . You can also post your question on the or the forum. If the matter is security related, please disclose it privately via --Issue: Many e2e are using optional condition fields such as Reason or Message to verify whether pod is failed or reached to the expected state having known expected reason and message. These are valid scenarios but we can not promote such e2e to Conformance as these Reasons and Messages may change over a period of time. This is observed mostly if we are expecting Pods/Jobs to get failed due to certain expected reason and message but failure may have tons of other reasons. References: 1) 2) 3) 4) 5) 6) [Conformance e2e] 7) [Conformance e2e] ........ So how such situations can be taken care while writing e2e? verifying status and message against non-empty field would resolve this issue but there would not be preciseness of e2e behavior._
/area conformance
What if we emphasize on using core libraries const for such reasons and messages, would it be fine? e.g.
No
/assign
Guidance on enum-like fields: Status fields: Old event guidance: Event guide: Events and contents of condition reasons and messages are not covered by API compatibility guarantees. They are passed through for human consumption, not machine consumption. Operational behaviors may be tuned by parameters, , or removed and not be considered breaking changes. For instance, when CrashLoopBackoff and ImagePullBackoff were introduced, they were new reasons not previously reported. Clients erroneously dependent on the previously reported reasons would no longer work. Similarly, when we change events to conform to the new style guide, their message text will change. Events may be aggressively garbage collected or even completely disabled in order to reduce apiserver load, and watching events is discouraged. We run old e2e tests against newer releases of Kubernetes. When those tests have been dependent on specific reasons, messages, and events, they have broken in the past. When contributors are encouraged to patch old e2e tests in order to pass version-skew testing, that should be a red flag that we are breaking backward compatibility.

We really need a policy that does not require significant technical analysis on a test-by-test basis. Tests that are dependent on such text may also indicate API deficiencies, or the tests may simply be testing what the code currently does rather than the API contract, and should be unit tests instead. I'll comment with case by case analysis in a bit.
is overly specific and shouldn't be checking the message. is overly specific and shouldn't be checking the reason. shouldn't be checking for events, or lack thereof. . 't be checking an event, and cluster autoscaling shouldn't be covered by conformance, anyway. 't be checking condition reason or message, or number of conditions. This is testing the implementation, not the required behavior. . As written, none of those tests are acceptable e2e tests, much less conformance tests.
I'm going to PR the main doc with a distillation of this and close this issue.
Note that this seems to be partly due to an organically evolved pattern to leave broken pods on nodes in order to work around bad controller behavior:


Question: package vhd import ( <del> "encoding/binary" </del> "encoding/hex" "fmt" "os" "strings" <del> "unicode/utf16" "unicode/utf8" </del> ) // https://groups.google.com/forum/#!msg/golang-nuts/d0nF_k4dSx4/rPGgfXv6QCoJ

I ran an OSS license scan of a codebase that has this project as a dependency and came across an interesting hit. This go-vhd dependency specified here: has a snippet of code from StackOverflow which is license under creative commons sharealike. Here's the code in question: I'm not a license expert, so I don't know if there are any real issues/conflicts here or not, but wanted to share to see what you all think. Here's SO's license info for reference: My understanding is that the potential issue would be with this clause: which could potentially require go-vhd and projects that depend on it to use the Creative Commons license as well.
Thanks for the report! can you please take a look if this is concerning? If so, we'll check if we can avoid this dependency (looks unlikely unless we duplicate what it's ) or we'll follow up upstream to update the code. /cc OWNERS for this repo
CC-BY-SA can be problematic in Apache-2.0 projects like k8s because of the ShareAlike aspect as noted. I'd recommend working with upstream to rewrite just that one function particularly since it looks like it's fairly limited.
let's please move this issue to kubernetes/kubernetes?
/provider azure
/sig release /area code-organization
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig release /area code-organization Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc /assign
/area code-organization
/assign
/milestone v1.19 /priority important-soon
Not a huge difference, but this is likely actually licensed under , not 4.0. See
the k/k code actually does not reference , since that function is in , do you mean we should remove from k/k totally?
Yes, we would either need to remove from k/k or ask the maintainers of to update . This means that (and k/k if we end up using it) would need to be licensed until CC-BY-SA.
Would this ultimately be backported to older versions of k8s or would you just do it in 1.19?

thanks, Let me remove the code in v1.19, actually that part of functionality is deprecated in Azure cloud provider already. My initial thought is only do this in v1.19
/kind bug /priority important-soon /sig cloud-provider /area provider/azure
Since it isn't trivial to remove as a dependency entirely (), I've created upstream to track updating the code there.


Question: if err := Convert_v1_AuthInfo_To_api_AuthInfo(&curr.AuthInfo, newAuthInfo, s); err != nil { return err } <ins> if *out == nil { *out = make(map[string]*api.AuthInfo) } </ins> if (*out)[curr.Name] == nil { (*out)[curr.Name] = newAuthInfo } else {

What happened: I have a test case: This behaves different with different client-go versions. With , the test passes. This is using in with in With , this test panics. This is using in with in Panic logs: Basically it its creating an object to decode into, and this object has some maps. These are set to nil, so when we try to populate them we get a panic. What you expected to happen: Behavior remains the same as previous release, or at least have some alternative that works here? How to reproduce it (as minimally and precisely as possible): I pasted the test above that reproduces this. This can be reproduced with a project with just that file, and a file with only client-go. Using go 1.13
/sig api-machinery
/assign


Question: return fmt.Errorf("Failed to delete destination for service %v, service not found", key.String()) } dests := f.Destinations[key] <del> var i int for i = range dests { if dests[i].Equal(dest) { </del> <ins> exist := false for i := range dests { if toRealServerKey(dests[i]).String() == toRealServerKey(dest).String() { // Delete one element f.Destinations[key] = append(f.Destinations[key][:i], f.Destinations[key][i+1:]...) exist = true </ins> break } } // Not Found <del> if i >= len(f.Destinations[key]) { </del> <ins> if !exist { </ins> return fmt.Errorf("Failed to delete real server for service %v, real server not found", key.String()) } <del> // Delete one element f.Destinations[key] = append(f.Destinations[key][:i], f.Destinations[key][i+1:]...) </del> return nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: IPVS real server can't be right deleted in FakeIPVS for ipvs testing. What you expected to happen: When delete ipvs real server, we can't give the weight information, so we can't use deep equal in .
/sig network /area ipvs
/kind bug


Question: }, { "ImportPath": "github.com/google/cadvisor/accelerators", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/api", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/cache/memory", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/client/v2", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/collector", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/common", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/containerd", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/crio", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/docker", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/libcontainer", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/raw", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/rkt", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/systemd", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/devicemapper", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/events", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/fs", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/healthz", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/http", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/http/mux", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v1", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v2", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/machine", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher/raw", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher/rkt", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/metrics", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages/static", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/storage", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/summary", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cloudinfo", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cpuload", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cpuload/netlink", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/docker", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/oomparser", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/sysfs", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/sysinfo", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/validate", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/version", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/cadvisor/zfs", <del> "Comment": "v0.28.3-20-g6116f265", "Rev": "6116f265302357cbb10f84737af30b1f13ce2d6c" </del> <ins> "Comment": "v0.29.0", "Rev": "aaaa65dba02880718d0237cd4e80ab8eb278bb19" </ins> }, { "ImportPath": "github.com/google/certificate-transparency/go",

/kind feature This is the once-per-release cut of a new cAdvisor release, which corresponds to the next kubernetes release. cAdvisor v0.29 corresponds to kubernetes v1.10. /sig node /priority important-soon
self-reminder to add per-cpu stats to ignore metrics when vendoring. reminder to add a release notice that one of the container labels fields is being removed.
/kind bug turns out most of the changes are bug-fixes
please mark this issue as approved-for-milestone, since it certainly seems to be, and it's now code slush. Thanks!


Question: }, expectedNeedsUpdate: true, }, <ins> { testName: "If service IPFamilies from single stack to dual stack", updateFn: func() { protocol := "http" oldSvc = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "tcp-service", Namespace: "default", }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{ Port:        80, Protocol:    v1.ProtocolTCP, TargetPort:  intstr.Parse("22"), AppProtocol: &protocol, }}, IPFamilies: []v1.IPFamily{v1.IPv4Protocol}, Type:       v1.ServiceTypeLoadBalancer, }, } newSvc = oldSvc.DeepCopy() newSvc.Spec.IPFamilies = []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol} }, expectedNeedsUpdate: true, }, { testName: "If service IPFamilies from dual stack to single stack", updateFn: func() { protocol := "http" oldSvc = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "tcp-service", Namespace: "default", }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{ Port:        80, Protocol:    v1.ProtocolTCP, TargetPort:  intstr.Parse("22"), AppProtocol: &protocol, }}, IPFamilies: []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol}, Type:       v1.ServiceTypeLoadBalancer, }, } newSvc = oldSvc.DeepCopy() newSvc.Spec.IPFamilies = []v1.IPFamily{v1.IPv4Protocol} }, expectedNeedsUpdate: true, }, { testName: "If service IPFamilies not change", updateFn: func() { protocol := "http" oldSvc = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "tcp-service", Namespace: "default", }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{ Port:        80, Protocol:    v1.ProtocolTCP, TargetPort:  intstr.Parse("22"), AppProtocol: &protocol, }}, IPFamilies: []v1.IPFamily{v1.IPv4Protocol}, Type:       v1.ServiceTypeLoadBalancer, }, } newSvc = oldSvc.DeepCopy() }, expectedNeedsUpdate: false, }, </ins> } controller, _, _ := newController()

Currently, when loadbalancer service's ipFammilies updatedservice controller will ignore this event. That means cloud provider can't update ipv4/ipv6 backend. So, I'm thinking service controller need add watching 'service's ipFammilies updated' event, It'd be important for cloud provider update dual-stack backend. code at Our load balancer supports dual-stack. When the services in the cluster are updated from IPv4 to IPv6(same ipv6-ipv4ipv4-ipv4/ipv6), it is expected that the cloud provider can perceive this event and update the IPv4/IPv6 address to the backend of the load balancer. /sig cloud-provider /area cloudprovider
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah, that is right, the logic need to be adapted /kind bug /sig network /sig cloud-provider