Question: return cc.Manager.Start() } <del> func (cc *cadvisorClient) exportHTTP(port uint) error { </del> <ins> func (cc *cadvisorClient) exportHTTP(address string, port uint) error { </ins> // Register the handlers regardless as this registers the prometheus // collector properly. mux := http.NewServeMux()

cAdvisor provides potentially sensitive data and there's currently no way to block access with anything other than iptables.
What type of cluster are you running (GKE, or what instructions did you use to setup your cluster?) A short term solution might be to try adding flag to kubelet and restarting the kubelet service and see if the sensitive information is no longer leaked. Doing this may the UI and heapster which expect to be able to poll the cadvisor port. I think probably knows if there is a longer term solution planned, such as: only serve on specified interface. if this is localhost, then it breaks the ui, but maybe that is okay. maybe has thought about this. require authentication/authorization to connect to cAdvisor port.
Self deployed cluster, the kubelet is started via
We are working on improving the kubelet APIs which will subsume cadvisor APIs. Until then setting cadvisor port to '0' is probably the way to go.
Is there an issue and target date for doing this?
Filed . The current plan is to complete this for v1.1. cc
Do you mean the read-only kubelet port ( by default IIRC)? If so, then we really need to disable this & provide authorisation options on the kubelet to be able to retrieve stats fro the kubelet via the normal port with a restricted token/cert/auth options. We've discussed this for OpenShift (openshift/origin) too.
Sorry I didn't realise that cadvisor was bound to port 4194 as well. Thought this was all embedded inside the kubelet process without exposing any ports.
Any reason why we start up cadvisor http server at all? Seems unnecessary now kubernetes embeds cadvisor & exposes via & ?
does not expose all the containers (, , etc). is working on introducing a better stats API in kubelet () that should help us shut down cadvisor's http server in kubelet.
Couldn't we just add cadvisor as a handler in the kubelet anyway? No need for separate http server & would mean access to raw data is available if anyone wants it.
Indeed that is an option. But we need a better stats API in kubelet anyways for reasons mentioned in .
Absolutely agree but thinking quick win.
+1 on a quick win.

We just disabled it by default instead: Which reminds me it would be great to have so we could make it easy to turn on/off stuff like this w/o rewriting the entire command-line in a service file.
cc
it's not 100% clear to me whether or not made it safe to use , could you please clarify? On a fresh k8s 1.6.4 cluster still exposes cAdvisor to the world: the file you referenced was deleted as part of coreos/coreos-overlay the documentation @ (CoreOS docs) doesn't mention cAdvisor should be disabled either. Another reason for my question to Vish.
Ahh. That's a bug that the new component config based scheme introduced. being is a valid setting.
I think we can change this. The KubeletConfiguration type is still technically alpha, so it should be ok to make this field a pointer and keep the current default when it's not provided (but allow 0). This won't affect anyone who presently omits the flag and expects port 4194. Theoretically people could be explicitly passing 0 and expecting 4194... I guess... but that's sort of an insane case, and I'd consider that behavior a bug.
Thanks for clarifying guys :+1: The original problem remains though: enabling the cAdvisor port makes the kubelet listen on all interfaces.
Yeah. We can get cAdvisor to bind to the interface that kubelet uses. cc , Antoine Cotten <:
/assign


Question: // start benchmark b.ResetTimer() <ins> // Start measuring throughput stopCh := make(chan struct{}) throughputCollector := newThroughputCollector(podInformer) go throughputCollector.run(stopCh) // Scheduling the main workload </ins> config = testutils.NewTestPodCreatorConfig() config.AddStrategy(testNamespace, test.PodsToSchedule.Num, testPodStrategy) podCreator = testutils.NewTestPodCreator(clientset, config) podCreator.CreatePods() <-completedCh <ins> close(stopCh) </ins> // Note: without this line we're taking the overhead of defer() into account. b.StopTimer() <ins> setNameLabel := func(dataItem *DataItem) DataItem { if dataItem.Labels == nil { dataItem.Labels = map[string]string{} } dataItem.Labels["Name"] = b.Name() return *dataItem } dataItems := []DataItem{ setNameLabel(throughputCollector.collect()), } for _, metric := range defaultMetrics { dataItem := newPrometheusCollector(metric).collect() if dataItem == nil { continue } dataItems = append(dataItems, setNameLabel(dataItem)) } return dataItems </ins> } func getPodStrategy(pc podCase) testutils.TestPodCreateStrategy {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: vmssNamesMap[vmSetName] = true } <ins> klog.V(2).Infof("ensureVMSSInPool begins to update VMSS %v with backendPoolID %s", vmssNamesMap, backendPoolID) </ins> for vmssName := range vmssNamesMap { vmss, err := ss.getVMSS(vmssName, azcache.CacheReadTypeDefault) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using Internal Standard LoadBalancer on Azure, although VMSS instances have been to LB backend address pools, the VMSS itself is not . Hence, if customer upgrades these VMSS instances to latest model, the backend address pool would be lost and hence the LoadBalancer services would be broken. What you expected to happen: Azure VMSS is also to internal SLB backend address pool automatically. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /kind bug /priority critical-urgent /area provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /priority critical-urgent /area provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
Can you share some insights when this could be get fixed?


Question: } } <ins> func TestServiceRegistryUpdateUnspecifiedAllocations(t *testing.T) { testCases := []struct { name  string svc   *api.Service // Need a clusterIP, NodePort, and HealthCheckNodePort allocated tweak func(*api.Service) }{{ name: "single-port", svc: svctest.MakeService("foo", svctest.SetTypeLoadBalancer, svctest.SetExternalTrafficPolicy(api.ServiceExternalTrafficPolicyTypeLocal)), tweak: nil, }, { name: "multi-port", svc: svctest.MakeService("foo", svctest.SetTypeLoadBalancer, svctest.SetExternalTrafficPolicy(api.ServiceExternalTrafficPolicyTypeLocal), svctest.SetPorts( svctest.MakeServicePort("p", 80, intstr.FromInt(80), api.ProtocolTCP), svctest.MakeServicePort("q", 443, intstr.FromInt(443), api.ProtocolTCP))), tweak: nil, }, { name: "shuffle-ports", svc: svctest.MakeService("foo", svctest.SetTypeLoadBalancer, svctest.SetExternalTrafficPolicy(api.ServiceExternalTrafficPolicyTypeLocal), svctest.SetPorts( svctest.MakeServicePort("p", 80, intstr.FromInt(80), api.ProtocolTCP), svctest.MakeServicePort("q", 443, intstr.FromInt(443), api.ProtocolTCP))), tweak: func(s *api.Service) { s.Spec.Ports[0], s.Spec.Ports[1] = s.Spec.Ports[1], s.Spec.Ports[0] }, }} for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { ctx := genericapirequest.NewDefaultContext() storage, server := NewTestREST(t, []api.IPFamily{api.IPv4Protocol}) defer server.Terminate(t) svc := tc.svc.DeepCopy() obj, err := storage.Create(ctx, svc.DeepCopy(), rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error: %v", err) } createdSvc := obj.(*api.Service) if createdSvc.Spec.ClusterIP == "" { t.Fatalf("expected ClusterIP to be set") } if len(createdSvc.Spec.ClusterIPs) == 0 { t.Fatalf("expected ClusterIPs to be set") } for i := range createdSvc.Spec.Ports { if createdSvc.Spec.Ports[i].NodePort == 0 { t.Fatalf("expected NodePort[%d] to be set", i) } } if createdSvc.Spec.HealthCheckNodePort == 0 { t.Fatalf("expected HealthCheckNodePort to be set") } // Update from the original object - just change the selector. svc.Spec.Selector = map[string]string{"bar": "baz2"} svc.ResourceVersion = createdSvc.ResourceVersion obj, _, err = storage.Update(ctx, svc.Name, rest.DefaultUpdatedObjectInfo(svc.DeepCopy()), rest.ValidateAllObjectFunc, rest.ValidateAllObjectUpdateFunc, false, &metav1.UpdateOptions{}) if err != nil { t.Fatalf("Expected no error: %v", err) } updatedSvc := obj.(*api.Service) if want, got := createdSvc.Spec.ClusterIP, updatedSvc.Spec.ClusterIP; want != got { t.Errorf("expected ClusterIP to not change: wanted %v, got %v", want, got) } if want, got := createdSvc.Spec.ClusterIPs, updatedSvc.Spec.ClusterIPs; !reflect.DeepEqual(want, got) { t.Errorf("expected ClusterIPs to not change: wanted %v, got %v", want, got) } portmap := func(s *api.Service) map[string]int32 { ret := map[string]int32{} for _, p := range s.Spec.Ports { ret[p.Name] = p.NodePort } return ret } if want, got := portmap(createdSvc), portmap(updatedSvc); !reflect.DeepEqual(want, got) { t.Errorf("expected NodePort to not change: wanted %v, got %v", want, got) } if want, got := createdSvc.Spec.HealthCheckNodePort, updatedSvc.Spec.HealthCheckNodePort; want != got { t.Errorf("expected HealthCheckNodePort to not change: wanted %v, got %v", want, got) } }) } } </ins> func TestServiceRegistryUpdateDryRun(t *testing.T) { ctx := genericapirequest.NewDefaultContext() storage, server := NewTestREST(t, []api.IPFamily{api.IPv4Protocol})

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: was closed as not a problem with kubectl, but with the server What you expected to happen: 'kubectl replace' with no change on a service should work. How to reproduce it (as minimally and precisely as possible): reproduction included in this issue Anything else we need to know?: Seems to me like this comment sums up the options I got here from this issue Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
After a long day of reading and experimenting with the Kubernetes API i found a work-around I want to share with you. Problem in my case: I'm using the Java Client K8 API, especially I got the exact same error like written in and here in this issue summary: I send an update of a service WITHOUT specifying the spec.clusterip within the V1Service object. The request gets then rejected with the reason . Workaround: First load the existing service that contains the current clusterIP. Set the old clusterIp to the updated V1Service. Also load the old metadata.resourceVersion value and set it to the new V1Service object, because leaving the value out using the Java API Client results in an error: Please do not close this issue due to inactivity again, like it was done with . Even if there are some workarounds, this issue will pop up again and again, because there really is a problem with the API restrictions or at least the messages you will get when interacting with the API. Constructive Suggestion: e.g. instead of throwing the error you could throw something like and for the resourceVersion:
Seriously, this has been going on for years. is over 2 years old and there have been dozens of bug reports/feature requests (in Kubernetes, Kubectl, Helm, and elsewhere) showing the problems the current behavior is causing. (List available on request.) is from over 3 years ago. is from 2015.

The old justifications for keeping the current behavior ("works as designed") have proven to be insufficient. It is time for a redesign. The API should accept as meaning "assign a clusterIP" on creation and "keep the current clusterIP" on update. Since, when you set on creation, the resource does not end up with , this means a resource cannot validly have , and since the field is immutable, could never be a correct value for modification under the current API. While this could be considered a breaking change, I doubt there is anyone who would complain about it. Rejecting as a modification is not helping anyone. It is not rejecting an accidental change or avoiding confusion because the meaning is unclear, and it is hard for me to imagine a use case where someone is relying on the current behavior and would be upset about the proposed new behavior (with the possible exception of people writing compliance tests). It is just causing problems for the sake of some kind of theoretical purity that is in the eye of the beholder. Treating as meaning "system should replace this with a valid setting" is equally pure.
cc long-term-issue (note to self)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
keepalive
Is there something I can do to move this forward?
/remove-sig cli /sig network /assign this is sort of the inverse of the issue fixed in Using with a static manifest on an object that has spec fields populated by the API server (like clusterIP in service) or controller (like replicas in deployment) is an anti-pattern. is a better match for this usage. Alternatively, a controller that did a 'read -modify -replace' could be ok, since it would be starting with the current API server state.
This is not impossible to fix in the PUT path (), but it basically turns PUT into PATCH. We have distinct PATCH operations for a reason, and that is almost certainly what you want in this case. should "just work" as a replacement for . Is there a reason why you can't use PATCH or apply?

- while I continue to argue that is wrong here, maybe we should just cave and make it quietly patch the value of clusterIP in this specific case? I really dislike it, not just for purity reasons, but because I have a hard time seeing if that will bite us later. Thoughts?
I really dislike it as well. "replace all spec content" is a bad thing to teach clients to do on resources that have fields set by different actors.
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. - My inclination is to close this. I know it's not what you want, but I do think you and most users will be better served by patch or apply in almost all cases (even knowing that Service has an ambiguity around changing ports that overlap each other which is not easily fixed because it is breaking). Is there a reason why you can't use patch or apply?
This really seems related to bug in Helm, as far as what I have read, and some recent (Jan2021) posts suggest that was fixed. Have not tested this out myself. Agree that adhering to expected PUT/PATCH api behavior would be best. Thanks for the followup!
I reject the idea that this is a bug in Helm, even though Helm is working hard (and failing) to handle the situation. I agree with you when you "Agree that adhering to expected PUT/PATCH api behavior would be best". My point is that the expected "PUT" behavior is that if a "PUT" request succeeds, the exact same "PUT" request given a second time should succeed as well. That is what is broken and I am asking to have fixed. Is there a better reason why I should be prohibited from using replace on a Service, when I can use it for every other resource? Why should Service require special-case handling by the user? I ask you to think of this in the larger context of using Infrastructure as Code (IaC) to manage resources, and the automation around that. In the IaC definition of a Service, the has to be set to in order to get a system-assigned address. Semantically, says "make the resource exactly like this". It can be, and is, used to ensure there is not "drift" in the configuration, i.e.

that the configuration stored in the IaC code base is live. It is the correct thing to use when you want to say "here is the configuration that should be active", and it is what automation uses to reset the configuration after it has drifted. Users should not have to treat Services specially, they should be able to any resources with the same spec that created them. Stating this another way, (A user should be able to the same thing repeatedly, successfully, and without causing changes after the first accepted .) By not accepting for in , you are breaking the contract of making idempotent. (As a matter of practicality, the Helm maintainers are trying to make up for the fact that is not idempotent, but that is not an excuse for Kubernetes to leave this API in its current state.) As I stated above, I am not even advocating for this case to be handled as a . As I see it, the value of is handled specially but not idempotently. The value of currently means one thing if the resource does not exist and something else if it does exist. I am asking instead that it consistently, idempotently be taken as meaning "any valid address: the existing one if one is set, otherwise one randomly assigned by the controller".
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. Unfortunately, to make PUT idempotent in this case, it would need to turn into a variant of PATCH. I'm not even saying I totally disagree, but that is what it means. As for PUT being idempotent, I don't think that's ever really been a hard objective. In this case (and look, I am not saying I am happy about it) that means reading existing state and patching it into the PUT. We can do that (probably) but I can't say for sure that there are no other APIs where this will happen (there probably are not). Note that to do this we would need to fixup , , and probably the newer dual-stack-related fields. Can someone help me understand the "bug"? Is it about using PUT instead of apply? I could see two paths to fixing this, if we want to (and I am still not 100%). 1) Change clusterIP and nodePort fields to pointers.

If not-specified on PUT AND the object exists AND the old warrants a value AND the new warrants a value, copy the existing values. This is a bit tricky for nodePort we'd have to at least specify what happens if the list is reordered or otherwise modified. For example, if the existing object has: and the update specifies: Do I track the move by name? What if the names changed, does that lose the tracking? We've had innumerable issues here because the API-specified merge-key was (is) ambiguous. The only way to get exactly what you want in ALL cases is to RMW the resource. Explicitly setting "" and 0 would work on initialization (compat) but would not be auto-patched. 2) Leave the fields as non-pointers and accept "" and 0 as "DWIM" mode. I dislike this because it makes it less obvious when a user mistakenly tries to clear a field - it used to fail and will now "succeed" but silently change the value. I am not sure I see a clever 3rd option. The only reason I entertain this at all is because of the issues around apply/patch and ports. NOTE: My willingness to think it thru is not a commitment to implement it. I am swamped right now :)
wrote I am shocked that you would think that, although I don't really know what you mean by "hard objective". The doc certainly implies should be idempotent, via such statements as and Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. But that stuff should already be handled by the more generalized resource handling architecture.

The general rule about PUT is that PUT on an existing resource should adjust that resource to be in the same working configuration as if you used that PUT to create a new resource. Calling PUT a second time with the same input should be acceptable and should not change the resource. I believe that Kubernetes generally has a canonical ordering of all fields in a resource (and if there is no clear way to order a list, the canonical order is simply the order in which the list was originally specified on resource creation), so what you do is enumerate the fields of the existing resource in canonical order, filter out the fields like that PUT cannot specify, and then compare it to the PUT resource, after normalizing it into canonical order, and add/delete/replace field by field. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . All we are asking for is that this case be interpreted differently and allowed. is documented as an allowable value in the input, but it is never an allowable value in the realized resource, so it cannot possibly mean to be interpreted as "force this value on the resource". Its meaning is that the system should assign a valid value and the user does not care what it is. So on PUT, when is matched against the exiting , he API should just confirm that is a valid value and, if so, accept the input.
As for PUT being idempotent, I don't think that's ever really been a hard objective. What I meant is it's just never been a primary question in reviewing APIs. Service is a weird case, far outside the norm and I'm not excusing it - just explaining what I meant. Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. For historical reasons, the "key" for ports was not the name but the port value, then the value + protocol. So we can key off that, I think.

I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. Those are all outside of . I don't know of any resource that PUT-PATCH inside the spec. Hilariously, I just noticed that nodePort is even worse - it accepts the non-value and then re-allocates the nodePort. I am going to call this a bug, but now we have a compatibility discussion to have. If a user does not specify a value, we can reasoinably argue that we should retain existing values. But if they specify , do we stay compatible and re-allocate the node port, or do we say that was always a bug? Is it a useful "feature" to be able to re-allocate? We disallow it explicitly gfor ClusterIPs. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. You haven't spent much time in the Service REST code, I guess. I have a large rework of that on-going, but this was not part of it. :) The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . Given the previous point about nodePort, would it be fair to modify your statement as: "when PUT says (or does not specify at all) ..." With the corollart that if you actually specify you will get the current (immutable error) behavior. This seems like it would satisfy helm and other users who are not specifying literal "" - they are not specifying any value at all. Thoughts?
At this point I am willing to call it a bug, albeit an unpleasant one.
wrote I have had regular run-ins with this issue for years, which is why I'm being insistent on getting it fixed properly. I do not recall having similar problems with and so I am less informed on those issues. I don't know what means in the spec or in practice, so I don't have a specific opinion as to what it should do to an existing resource.

I will simply restate the general principle which should apply to just like everything else: what ever you allow as a setting on creation should be allowed in any and should not cause a change if repeatedly applied. The very practical reason for this is so that I can record some place (like a checked into a VCS) that "this is how the resource should be configured" and use that (1) to create the resource and (2) to return the resource to expected configuration after some external operation changed it, using a standard API while otherwise treating the resource as opaque.
/cc
/cc (i know i am going to regret this)
It is a bug. Not because it does not match rest api semantics. But because it is not consistent with the rest of the apis. In other words, If user can do on other resources they should be able to do it on as well. It is hard (and weird) to ask users to do one thing for all resources, but then do a totally different thing for Services. As discussed over sig-network the problem will be user intent. Some ideas for a solution: If irrespective of changes in other we can assume that the user is replacing and does not care about ClusterIP value, hence copying to new should be fine. It is basically like the user saying so copying old=new should be Ok. If that is a failure case, because it breaks ClusterIP immutability rules ( we should really, really remove ClusterIP immutability but i digress). We will still require the validation to work as is. e.g. replacing with with a type that does not require ClusterIP and not setting it to should fail. I can look into building a POC for this.
I am probably missing some subtlety here, but as in I suspect that Helm is behaving properly and the problem is in way Kubernetes handles s: the of a resource should never be rewritten with variable environmental information; that is what is for. In other words could be set in the unusual case that you wished to pick an IP yourself but would normally be null; would reflect the service’s current value, expected to be retained across unrelated patches.
nailed it on the head with his last comment. I do agree that we should not be re-defining the behaviour of a PUT request.

But the inconsistency of how a Service updates the field compared to other resources results in inconsistencies in the PUT behaviour. Once a resource defines fields in the object, normally Kubernetes leaves it unchanged. My understanding is that Kubernetes uses this field to use as a reference point to be the "ideal state of the world", using the field to record the "current state of the resource". The only offending case I've seen in the wild is a , as it updates both the field AND the field. The end result is that a with a defining results in differing behaviour than the rest of the resource APIs. From Helm's point of view, a chart author's only course of action is to a) omit from the manifest and all subsequent updates, or b) if you happened to define in your first install, find the value Kubernetes provided and inject it into your template so that the value Kubernetes provided is preseved. Failing to do so results in a failed PUT () request. I wrote this behaviour down in-depth in this Helm comment: I'd propose an alternative solution than solutions: instead of updating after an IP has been assigned to the Service, use the API, i.e. . It would be a breaking change to the API, but it should bring Services back in line with the rest of the resource APIs, leaving as a user-defined field.
I agree
I'm afraid that breaking an API is not a feasible option
That document does allow for breaking changes under the apiVersion. Instead of making a breaking change to v1 services, you can introduce Service v2. Existing v1 services migrate their spec.clusterIP’s existing value (auto-generated or not) to the new version. spec.clusterIP and status.clusterIP would now match their corresponding relationships, and Kubernetes retains backwards/forwards compatibility.
(If it's not clear whether the API changes document does or doesn't allow the change from last comment, let's raise a separate improvement request against that document).
Does the case is important ? Because here I can read or ...
I have a patch. I'll try to get it out for review tonight.
Has anyone yet checked that the purported fix actually addresses the issue in Helm?
Please do!
I can still see the error but I'm on GKE 1.18, will this only apply to Kubernetes 1.21/22?

It will be part of 1.22 or the next 1.21.X (I don't remember now what is the latest stable)
Can anyone confirm that this issue was actually fixed in k8s version 1.21.4 or 1.22.1 ? I do not see it listed in the release notes/change log
The change was released in the 1.22.0-beta.2, so yes it should be available in 1.22. See I do not believe it was backported to 1.21.
Thanks for confirming and providing the details


Question: var map_JobStatus = map[string]string{ "":               "JobStatus represents the current state of a Job.", <del> "conditions":     "The latest available observations of an object's current state. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/", </del> <ins> "conditions":     "The latest available observations of an object's current state. When a job fails, one of the conditions will have type == "Failed". More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/", </ins> "startTime":      "Represents time when the job was acknowledged by the job controller. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC.", <del> "completionTime": "Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC.", </del> <ins> "completionTime": "Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC. The completion time is only set when the job finishes successfully.", </ins> "active":         "The number of actively running pods.", "succeeded":      "The number of pods which reached phase Succeeded.", "failed":         "The number of pods which reached phase Failed.",

The API docs for jobs are unclear as to what fields a caller would use to determine job success or failure in 1.11 (and we haven't updated them in a while) A casual reader would infer that completionTime is set when the job fails (but it isn't). Linking out to the docs is ok, but I shouldn't have to click through the links to understand how to use the Job object. The Job description and the job status fields should clearly indicate what fields are set in completion states on the job, because that's 100% of the point of docs. Taking a summary of the jobs-run-to-completion link and putting it in , , and job would improve the actual utility of our API doc. We should also do a pass on other objects and make the actual in server docs more useful.
as a consumer of the API, I had to read way too much to know how to use jobs correctly and how to write an API client that properly handled the conditions. Also, we should consider either setting completionTime on failures, or adding failedTime (the former may not be forwards compatible), because reading the conditions is too complex for a fundamental use case like this. This matches our decision w.r.t. conditions vs. fields - conditions are useful for extension and extra data, but status should have fields that allow an API consumer to easily understand state.
Also... our docs don't actually explain how to determine the job is complete via the API, so there's arguably no place this is captured.
I have the same problem right now, of correctly identifying a finished job, when it does not complete successfully and after reading the documentation and the reference and all the examples I could find, it's still not clear how I can check the conditions and make sure the job is finished when I do so: -what does each mean? can I rely on , is that set only when the job finishes, whether via or ? what other reasons are possible? It would definitely help to have specified on failure as well, and the differentiation to be made via the presence of .
/assign soltysh
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I have to say this issue is still very relevant. As of now, it is very hard to design something around Job Statuses. Anyway to move this forward?
Completely agree with previous comments! Even I found the way how to detect completed/failed jobs, still not sure am I covered all use cases. This part definitely needs clarifications/improvements/better documentation.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
With regards to: If you actually follow this link and then search on that page for "conditions" you will get exactly zero hits. This is extremely unhelpful.
This documentation really, really needs work. (So does the API, for that matter.)
So what is the correct way to determine if a job is finished and whether it completed successfully?
When a Job completes (finished successfully), its will be set, and Job conditions will have one with . When a Job fails, Job conditions will have one with .
Hi, if nobody has been working on it, can I take this one? I have gone through the issue which is mentioned here and I would like to work on this. Can you please help me to get started as I am beginner to the project.
I totally agree with this- currently the only way to get the full picture of a Job is to read its conditions along with status fields which is error prone and a lot of extra work. I think a good start would be to add a new field as suggested by so as to not break compatibility for users already depending on
This sound reasonable. I'm still unclear when can contain more than one element. Also what does a or mean?
Still, open -- anyone actively working on this? I would like to attempt if not. :)
Are you still working on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
bump
This does not match what I just witnessed in my cluster… digging deeper to see if I need to file a bug but I have a Job with a single Pod that was executed. The Pod failed but the Job was marked as complete. This is the one (and only) pod that executed as my job: And this is the job that owns it (from kubectl describe): This is the job status object: on v1.16.7 but not finding any similar reports. just found this page as my only relevant search result.
I think it not only needs documentation, but also a change: It would be good to have 2 fields (probably needs better names), this way it's easy and clear when a job is finished and if it failed or not and when (someone can look at failed, succeeded and conditions for details): finishedtime: only set when the job finished (either completely failed or completely finished) finishtype: 'failed or completed'


Question: expected: "unknown", }, } <del> for i, data := range cases { syncPodString := data.sp.String() assert.Equal(t, data.expected, syncPodString, "test[%d]", i) t.Logf("Test case [%d]", i) </del> <ins> for _, test := range tests { t.Run(test.expected, func(t *testing.T) { syncPodString := test.sp.String() assert.Equal(t, test.expected, syncPodString) }) } } func TestIsMirrorPod(t *testing.T) { tests := []struct { name     string pod      *v1.Pod expected bool }{ { name:     "mirror pod", pod:      getTestPod(configMirrorAnnotation(), nil), expected: true, }, { name:     "not a mirror pod", pod:      getTestPod(nil, nil), expected: false, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { isMirrorPod := IsMirrorPod(test.pod) assert.Equal(t, test.expected, isMirrorPod) }) } } func TestIsStaticPod(t *testing.T) { tests := []struct { name     string pod      *v1.Pod expected bool }{ { name:     "static pod with file source", pod:      getTestPod(configSourceAnnotation(FileSource), nil), expected: true, }, { name:     "static pod with http source", pod:      getTestPod(configSourceAnnotation(HTTPSource), nil), expected: true, }, { name:     "static pod with api server source", pod:      getTestPod(configSourceAnnotation(ApiserverSource), nil), expected: false, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { isStaticPod := IsStaticPod(test.pod) assert.Equal(t, test.expected, isStaticPod) }) } } func TestIsCriticalPod(t *testing.T) { tests := []struct { name     string pod      *v1.Pod expected bool }{ { name:     "critical pod with file source", pod:      getTestPod(configSourceAnnotation(FileSource), nil), expected: true, }, { name:     "critical pod with mirror annotation", pod:      getTestPod(configMirrorAnnotation(), nil), expected: true, }, { name:     "critical pod using system priority", pod:      getTestPod(nil, &systemPriority), expected: true, }, { name:     "critical pod using greater than system priority", pod:      getTestPod(nil, &systemPriorityUpper), expected: true, }, { name:     "not a critical pod with api server annotation", pod:      getTestPod(configSourceAnnotation(ApiserverSource), nil), expected: false, }, { name:     "not critical if not static, mirror or without a priority", pod:      getTestPod(nil, nil), expected: false, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { isCriticalPod := IsCriticalPod(test.pod) assert.Equal(t, test.expected, isCriticalPod) }) } } func TestPreemptable(t *testing.T) { tests := []struct { name      string preemptor *v1.Pod preemptee *v1.Pod expected  bool }{ { name:      "a critical preemptor pod preempts a non critical pod", preemptor: getTestPod(configSourceAnnotation(FileSource), nil), preemptee: getTestPod(nil, nil), expected:  true, }, { name:      "a preemptor pod with higher priority preempts a critical pod", preemptor: getTestPod(configSourceAnnotation(FileSource), &systemPriorityUpper), preemptee: getTestPod(configSourceAnnotation(FileSource), &systemPriority), expected:  true, }, { name:      "a not critical pod with higher priority preempts a critical pod", preemptor: getTestPod(configSourceAnnotation(ApiserverSource), &systemPriorityUpper), preemptee: getTestPod(configSourceAnnotation(FileSource), &systemPriority), expected:  true, }, { name:      "a critical pod with less priority do not preempts a critical pod", preemptor: getTestPod(configSourceAnnotation(FileSource), &systemPriority), preemptee: getTestPod(configSourceAnnotation(FileSource), &systemPriorityUpper), expected:  false, }, { name:      "a critical pod without priority do not preempts a critical pod without priority", preemptor: getTestPod(configSourceAnnotation(FileSource), nil), preemptee: getTestPod(configSourceAnnotation(FileSource), nil), expected:  false, }, { name:      "a critical pod with priority do not preempts a critical pod with the same priority", preemptor: getTestPod(configSourceAnnotation(FileSource), &systemPriority), preemptee: getTestPod(configSourceAnnotation(FileSource), &systemPriority), expected:  false, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { isPreemtable := Preemptable(test.preemptor, test.preemptee) assert.Equal(t, test.expected, isPreemtable) }) </ins> } } func TestIsCriticalPodBasedOnPriority(t *testing.T) { tests := []struct { <del> priority    int32 description string expected    bool </del> <ins> priority int32 name     string expected bool </ins> }{ { <del> priority:    int32(2000000001), description: "A system critical pod", expected:    true, </del> <ins> name:     "a system critical pod", priority: systemPriority, expected: true, </ins> }, { <del> priority:    int32(1000000000), description: "A non system critical pod", expected:    false, </del> <ins> name:     "a non system critical pod", priority: scheduling.HighestUserDefinablePriority, expected: false, </ins> }, } for _, test := range tests { <del> actual := IsCriticalPodBasedOnPriority(test.priority) if actual != test.expected { t.Errorf("IsCriticalPodBased on priority should have returned %v for test %v but got %v", test.expected, test.description, actual) } </del> <ins> t.Run(test.name, func(t *testing.T) { actual := IsCriticalPodBasedOnPriority(test.priority) if actual != test.expected { t.Errorf("IsCriticalPodBased on priority should have returned %v for test %v but got %v", test.expected, test.name, actual) } }) </ins> } }

What would you like to be : Currently, in the following public methods do not have unit test coverage: - - These methods are great candidates for adding unit test coverage. Why is this needed: Adding unit test coverage will both ensure they are working correctly and (more importantly), encode their expected behavior in the test suite. Additionally, if we want to change their behavior in the future, it'll be helpful to already have the test suites. Extra note I think this could be a great candidate for someone who is new to contributing to sig-node/k8s. I'm happy to mentor any first time contributors :) Please respond saying you'd like to work on this issue and I'll assign it to you. Please reach out to me on k8s slack (where I'm also if I can be of any assistance.
/sig node
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I would like to work in one of these.
/assign Awesome!
Hi, I am a first time contributor and would be happy to contribute to one of these.
Hi, I want to help if still need it.


Question: return err } <del> // WaitForPodAddition waits for pods to be added within the timeout. func WaitForPodAddition(c clientset.Interface, ns string, timeout time.Duration) error { options := api.ListOptions{FieldSelector: fields.Set{ "metadata.namespace": ns, }.AsSelector()} w, err := c.Core().Pods(ns).Watch(options) if err != nil { return err } _, err = watch.Until(timeout, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Added: return true, nil } Logf("Waiting for pod(s) to be added in namespace %v", ns) return false, nil }) return err } </del> func WaitForPodToDisappear(c clientset.Interface, ns, podName string, label labels.Selector, interval, timeout time.Duration) error { return wait.PollImmediate(interval, timeout, func() (bool, error) { Logf("Waiting for pod %s to disappear", podName)

https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
can you please take a look at this test or reassign it to someone. It is becoming much more frequent and is failing the serial test suite preventing me from finding a green build to branch for 1.5.
Looks like you've touched this test in the past. Could you take a look at it why it is flaking so frequently now?
This is likely due to the NC changes introduced in I remember having to touch that test. I'll take a look at this today.
Thank you!

I'm trying to cut the 1.5 branch and I can't find a green build so the sooner this can get fixed the better!


Question: cpuLimit := container.Resources.Limits.Cpu() if !cpuLimit.IsZero() { <del> // Note that sysinfo.NumCPU() is limited to 64 CPUs on Windows due to Processor Groups, // as only 64 processors are available for execution by a given process. This causes // some oddities on systems with more than 64 processors. // Refer https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx. </del> // Since Kubernetes doesn't have any notion of weight in the Pod/Container API, only limits/reserves, then applying CpuMaximum only // will better follow the intent of the user. At one point CpuWeights were set, but this prevented limits from having any effect.

When submitting a windows pod using resource limits on a node with more than one processor group, the specified count is incorrect due to the usage of which does not take processor groups into account. For example: node with 96 cores cpu limit 94 returns This results in a CPU maximum of since the range is from [1, ]: Our initial goal for this was to limit a process to not use all available CPU and therefore render the node uncontrollable. A CPU maximum of instead of for that specific case. For the calculation, it should use the CPU count for all available processor groups instead of just the current one. Launch a node with a lot of CPUs, for example and schedule the following pod on it: Then SSH into the node and inspect the running container using . The value will be set to . (which just uses ) is able to correctly find out the number of CPUs available to a system. Using for calculation or incorporating that code might solve this issue. <details</details<detailsAzure in-tree </details<details</details<details</details<detailscontainerd://1.6.10 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
There is already some code which uses GetActiveProcessorCount:
/assign
/triage accepted


Question: IgnoreNotFound: &ignoreNotFound, Now:            &now, Timeout:        &timeout, <ins> Wait:           &wait, </ins> Output:         &output, } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc


Question: expectReason: `seLinuxOptions`, expectDetail: `containers "d", "e", "f" set forbidden securityContext.seLinuxOptions: type "bar"; user may not be set; role may not be set`, }, <ins> { name: "bad type", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ SELinuxOptions: &corev1.SELinuxOptions{ Type: "bad", }, }, }}, expectReason: `seLinuxOptions`, expectDetail: `pod set forbidden securityContext.seLinuxOptions: type "bad"`, }, { name: "bad user", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ SELinuxOptions: &corev1.SELinuxOptions{ User: "bad", }, }, }}, expectReason: `seLinuxOptions`, expectDetail: `pod set forbidden securityContext.seLinuxOptions: user may not be set`, }, { name: "bad role", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ SELinuxOptions: &corev1.SELinuxOptions{ Role: "bad", }, }, }}, expectReason: `seLinuxOptions`, expectDetail: `pod set forbidden securityContext.seLinuxOptions: role may not be set`, }, </ins> } for _, tc := range tests {

If I configure the namespace to enforce PSS checks, the return message ends abruptly for : The resource manifest that can be used to reproduce:
/transfer kubernetes
/sig auth /cc
Thanks for the report, fix in
Thanks for the quick fix !


Question: //  * If the request can't be constructed, or an error happened earlier while building its //    arguments: *RequestConstructionError //  * If the server responds with a status: *errors.StatusError or *errors.UnexpectedObjectError <del> //  * If the status code and body don't make sense together: *UnexpectedStatusError </del> //  * http.Client.Do errors are returned directly. func (r *Request) Do() Result { start := time.Now() defer func() { metrics.RequestLatency.WithLabelValues(r.verb, r.finalURLTemplate()).Observe(metrics.SinceInMicroseconds(start)) }() <del> body, err := r.DoRaw() </del> <ins> var result Result err := r.request(func(req *http.Request, resp *http.Response) { result = r.transformResponse(resp, req) }) </ins> if err != nil { return Result{err: err} } <del> respBody, created, err := r.transformResponse(r.resp, r.req, body) return Result{respBody, created, err, r.codec} </del> <ins> return result </ins> } <del> // transformResponse converts an API response into a structured API object. If body is nil, the response // body will be read to try and gather more response data. func (r *Request) transformResponse(resp *http.Response, req *http.Request, body []byte) ([]byte, bool, error) { if body == nil && resp.Body != nil { </del> <ins> // DoRaw executes the request but does not process the response body. func (r *Request) DoRaw() ([]byte, error) { start := time.Now() defer func() { metrics.RequestLatency.WithLabelValues(r.verb, r.finalURLTemplate()).Observe(metrics.SinceInMicroseconds(start)) }() var result Result err := r.request(func(req *http.Request, resp *http.Response) { result.body, result.err = ioutil.ReadAll(resp.Body) }) if err != nil { return nil, err } return result.body, result.err } // transformResponse converts an API response into a structured API object func (r *Request) transformResponse(resp *http.Response, req *http.Request) Result { var body []byte if resp.Body != nil { </ins> if data, err := ioutil.ReadAll(resp.Body); err == nil { body = data }

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: CLUSTER_DIR?=$(shell pwd)/../../../cluster/ <del> BASEIMAGE=debian:stretch-slim </del> <ins> BASEIMAGE=debian:stable-slim </ins> TEMP_DIR:=$(shell mktemp -d -t conformanceXXXXXX) all: build

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: produces an error: What you expected to happen: Conformance tests should be launched. Examining image contents indicates executables are of the correct architecture. How to reproduce it (as minimally and precisely as possible): From aarch64 machine: Anything else we need to know?: Works with v1.17.4: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Architecture /sig Testing
/sig release /remove-sig architecture
any thoughts on this issue at all? We use this to test K3d and are blocked on releasing anything in the 1.18 line because of it.
I don't have expertise in the image release pipeline, I'd suggest reaching out to
Hi Did you run this test in an arm64 virtual machine? If so, I have fixed this issue in cadvisor side. This issue is caused by cadvisor. Currently, the version of cadvisor in kubernetes is v0.36.1-0.-. There is a defect in this version, it gets the wrong machine info and reports num_cores=0 on Arm64 guest. I have delivered the PR into cadvisor in 22 days ago. And this patch was merged in 10 days ago. Please see my pr as reference: Currently, I'm waiting a new tag of cadvisor, so that I can use this new tag in kubernetes and fix this issue in k8s side. Please see my issue: Thanks.
Currently we only have one Arm64 conformance test is alive, this conformance test was run on 2 AWS arm64 instances https://k8s- All other VM-based Arm64 conformance tests are down.
/area conformance
Thanks for taking a look just to be sure it is not a VM issue I fired up an a1-metal arm64 aws instance.

Please note I am using sonobuoy for running tests rather than kubetest, I tried to find if kubetest is using the conformance image for testing and from a quick check maybe not. The error is not in the conformance test itself, but launching the conformance test image on arm64. The is . Tried again with with 1.18.2 and got the same error: still works ok for the v1.17 branch: It looks like the build chain for images has changed: So there are now six less layers of the conformance image for the 1.18 branch. Please also note how the arm64 images show amd64 as the architecture for both branches. This has been problematic for trying to air-gap arm64 properly and is a systemic issue for many of the projects on gcr (and elsewhere).
Hi I see. I'll check it. Thanks.
/reopen This issue is still unfixed as of today. here is a reproducer that helps doing the reverse test on a x8664 machine: So this is an image where the base os is actually arm64. starting with v1.18.0 and still as of today (v1.18.5-rc.0 and v1.19.0-beta.2) the uploaded -arm64 image is actually an amd64 image. There must be a typo somewhere. testable this way: This is supposed to generate the 'exec file error', but it doesn't, which indicates that it is actually an x8664 image (amd64).
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen x8664 # podman pull # podman run 66652b9ed7f16a9bc848038ddb00159f3e2d22d8c2bb1ba050913309daf101c9 cat /etc/os-release exec user process caused "exec format error" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC the fix was reverted due to not working in the release pipeline. I did say that would hurt :upsidedownface:
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten This is still blocking conformance testing on arm64, so I hope it can be considered open.
Looks like to me this is a very complicated issue which is related with GCB, Moby Buildkit, it's not easy for me to reproduce and some trouble shooting by myself since I am not able to access GCB. Just wanna know is there any follow up to make this PR merge again? , we do have some needs to run the conformance testing on ARM64 .
SIG testing does not use this image (we prefer not to run the test runner as a pod given our clusters may be unreliable since we're testing changes and we want to avoid e.g. the pod being evicted =data loss), however this is currently used by the CNCF conformance program (which is CNCF / SIG arch) and sonobuoy (which is a vmware project). A bunch of our images are shipping with buildkit / buildx currently at HEAD, something like number could probably be brought back if someone wanted to work on this. Currently I'm not going to have time myself.
/cc
cc for arm64 image :(
xref /assign
It looks like that the issue is not in k/release, since we only build/stage the images via . Then we push the images and annotate them to their corresponding architecture.

Looking at the staged artifacts, for example for v1.20.1 arm64: Then we can see that also kube-proxy seems to have this issue: I assume that the fix is somewhere in the build scripts, where I'm now looking into. Beside that, all other architectures (arm, ppc64le, ...) are affected by this bug as well (for the same images only). Edit: Looks like the distroless images are fine, whereas the debian based images (proxy / conformance) are pointing to the wrong architecture. Conformance seems to always use , which might be the issue here. Edit: Done with my experiments. will change the already open PR according to our findings from the slack conversation (), whereas we have to change the used base images for kube-proxy to match the corresponding target architecture (like using for )


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "fmt" "time" federation_release_1_4 "k8s.io/kubernetes/federation/client/clientset_generated/federation_release_1_4" pkg_runtime "k8s.io/kubernetes/pkg/runtime" ) // Type of the operation that can be executed in Federated. type FederatedOperationType string const ( OperationTypeAdd    = "add" OperationTypeUpdate = "update" OperationTypeDelete = "delete" ) // FederatedOperation definition contains type (add/update/delete) and the object itself. type FederatedOperation struct { Type FederatedOperationType Obj  pkg_runtime.Object } // A helper that executes the given set of updates on federation, in parallel. type FederatedUpdater interface { // Executes the given set of operations within the specified timeout. // Timeout is best-effort. There is no guarantee that the underlying operations are // stopped when it is reached. However the function will return after the timeout // with a non-nil error. Update([]FederatedOperation, time.Duration) error } // A function that executes some operation using the passed client and object. type FederatedOperationHandler func(federation_release_1_4.Interface, pkg_runtime.Object) error type federatedUpdaterImpl struct { federation FederationView addFunction    FederatedOperationHandler updateFunction FederatedOperationHandler deleteFunction FederatedOperationHandler } func NewFederatedUpdater(federation FederationView, add, update, del FederatedOperationHandler) FederatedUpdater { return &federatedUpdaterImpl{ federation:     federation, addFunction:    add, updateFunction: update, deleteFunction: del, } } func (fu *federatedUpdaterImpl) Update(ops []FederatedOperation, timeout time.Duration) error { done := make(chan error, len(ops)) for _, op := range ops { go func(op FederatedOperation) { clusterName, err := GetClusterName(op.Obj) if err != nil { done <- err return } // TODO: Ensure that the clientset has reasonable timeout. clientset, err := fu.federation.GetClientsetForCluster(clusterName) if err != nil { done <- err return } switch op.Type { case OperationTypeAdd: err = fu.addFunction(clientset, op.Obj) case OperationTypeUpdate: err = fu.updateFunction(clientset, op.Obj) case OperationTypeDelete: err = fu.deleteFunction(clientset, op.Obj) } done <- err }(op) } start := time.Now() for i := 0; i < len(ops); i++ { now := time.Now() if !now.Before(start.Add(timeout)) { return fmt.Errorf("failed to finish all operations in %v", timeout) } select { case err := <-done: if err != nil { return err } case <-time.After(start.Add(timeout).Sub(now)): return fmt.Errorf("failed to finish all operations in %v", timeout) } } // All operations finished in time. return nil } </ins>

Federated controllers will oftentimes have to crreate/update/delete the same/similar structures in multiple clusters. Given that the roundtrip time to remote clusters can be high (like, lets say, 1 sec) these operations should be done in parallel so that the main controller goroutine is not blocked for long. Part of


Question: imageService internalapi.ImageManagerService // logMetrics provides the metrics for container logs logMetricsService LogMetricsService <ins> // cpuUsageCache caches the cpu usage for containers. cpuUsageCache map[string]*runtimeapi.CpuUsage mutex         sync.Mutex </ins> } // newCRIStatsProvider returns a containerStatsProvider implementation that

Related: On Windows, kubelet reports only the cumulative for cpu usage. On Linux, kubelet reports both and the instantaneous . Although In the future, most components/monitoring pipelines will switch to using the cumulative metrics, a few features still depend on the instantaneous metrics today. Examples are: , HPA/VPA. Kubelet should report this metrics on Windows to support the existing features. /cc who explained to me what features depend on the cpu metrics. /cc /cc
- is this something that's clear enough in the CRI layer that you can implement it, or do we need to fix that first?
/assign
do you have any notes or a doc you can point to that better describes what features depend on which metrics? It would be good for us to share and see if anything other than is broken
The horizontal and vertical pod autoscalers also depend on the
Documentation on metrics API and metrics server (i.e., the aggregator): The metrics-server pod keeps spewing errors like this: Running : I haven't verified whether pod autoscaling works in my cluster, but I assume they won't since metrics server seems to discard the metrics from the windows pods. I believe the implementation is in kubelet (for the docker integration), so this should not need CRI changes.
I think this is generally an issue for non-Docker CRIs (not necessarily Windows-exclusive). The kubelet source code suggests that these metrics will be gathered from the CRI (if the CRI is not Docker). The struct that kubelet expects returned from the ContainerStats RPC does not include usageNanoCores. Does this mean that, until this is resolved, custom CRIs are unable to satisfactorily report metrics to the 'metrics-server'?
- are you able to work on this?
that is a good point. For CRI runtimes, the kubelet should calculate the average over 10s to produce this metric.
Good point. We designed CRI around core metrics API, but some CRI runtimes have integrated with cadvisor directly to fill the gap while we wait for core metrics API. I think it's reasonable for kubelet to do this on behalf of all CRI runtimes. I spoke to offline, and this would probably require no more than 2 days of work. If sig-windows (e.g., can pick this up, that'd be great.

has agreed to review the changes (thanks!) If not, we should clearly document that this is not supported in Windows GA for 1.14.
thanks for the update. we will wait to hear back from
Currently, is got from docker's stats API on Windows. From docker stats API , only is available, hence is not set on Windows: Do you know any way to get instantaneous for Windows containers?
I have no idea. any idea?
I think this is broader than just a Windows issue. Officially supported tools such as the kubernetes metrics-server expect the field, but custom CRIs that conform to the interfaces defined in the package are not given the liberty to return this value.
suggested previously to cache the stats temporarily and compute the stats using the diff. David, could you chime in on how this could be done? Yes, this was mentioned in the previous comments too. There are plans to transition those components off this field, but it's WIP. We are looking for a workaround in kubelet for now.
my biggest worry is that if we say this impacts the horizonal POD autoscaler, how is that passing in our end2end and conformance tests?
Do you test HPA? It's not part of conformance, and I doubt this is included in windows testing.
As to how this could be done, in and we could just add a in the struct, which we assign the previous CpuUsage metric for each container. Then, when we need to calculate the rate, we pass the previous CpuUsage along with the current one to , and calculate a rate based on the timestamp and values.
ListPodCPUAndMemoryStats() we could just add a map[string]*runtimeapi.CpuUsage in the criStatsProvider struct I've done some tinkering on this myself and I originally thought that it would be as simple as that. It gets much more complicated when you have to work in between the existing cAdvisor-based logic. I assume the intent is to cut it out entirely eventually, but the information passed through the CRI doesn't even come close to being so comprehensive. So far I've found that this approach fixes the problem for the "/stats/summary" api, but not necessarily for the other apis. In particular, "/stats/container" seems to be completely based on cAdvisor, being unimpacted by the changes we made to those two functions.

Some monitoring tools and metrics scrapers rely on the "/stats/container" api for their metrics, which means it is insufficient to update one api without doing the other.
We don't need to fill in everything in the summary API, just "usageNanoCores". The "/stats/summary" api is the one used by the metrics server, which is why we care about it in this case. The "/stats/container" isn't something we need to implement to support this.
we a couple of tickets to validate if HPA for Windows containers works. My guess is that it works for memory and not for CPU metrics because of this issue. If this is complicated to fix, i wonder if waiting for the CRI containerD changes in windows would be the best option for supporting this.
containerd and cri-o bypass this issue by leveraging cadvisor as a temporary solution. I don't think that'd work for windows.
Sorry the for late response. HPA is actually working for Windows containers. e.g. in my cluster (v1.12.5), cpu-based HPA is working as expected: Which version of kubernetes are you referring for HPA issues?
are you running heapster, or the metrics server to serve HPA? Heapster uses the cumulative metric, but the metrics-server uses the 10 average rate metric.
metrics-server is used in my cluster.
See .
/milestone v1.14


Question: // Check whether timeout has reached the maximum waiting time timeout := elapsedTime > rc.maxWaitForUnmountDuration <ins> isHealthy, err := rc.nodeIsHealthy(attachedVolume.NodeName) if err != nil { klog.Errorf("failed to get health of node %s: %s", attachedVolume.NodeName, err.Error()) } // Force detach volumes from unhealthy nodes after maxWaitForUnmountDuration. forceDetach := !isHealthy && timeout </ins> hasOutOfServiceTaint, err := rc.hasOutOfServiceTaint(attachedVolume.NodeName) if err != nil { klog.Errorf("failed to get taint specs for node %s: %s", attachedVolume.NodeName, err.Error()) } <del> // Check whether volume is still mounted. Skip detach if it is still mounted unless timeout </del> <ins> // Check whether volume is still mounted. Skip detach if it is still mounted unless force detach timeout </ins> // or the node has `node.kubernetes.io/out-of-service` taint. <del> if attachedVolume.MountedByNode && !timeout && !hasOutOfServiceTaint { </del> <ins> if attachedVolume.MountedByNode && !forceDetach && !hasOutOfServiceTaint { </ins> klog.V(5).InfoS("Cannot detach volume because it is still mounted", "volume", attachedVolume) continue }

A/D controller tries to detach volumes that are still mounted after 6 minutes. The conditions to trigger the detach are: I.e. all containers are finished. But that does not guarantee that volumes are unmounted. When a CSI driver can't unmount a volume for 6 minutes, the volume is force detached by A/D controller, corrupting the volume and leaving volume mounts on the node in a bad state. Note that the node is perfectly healthy, it just can't unmount a volume from whatever reason. A/D controller detaches volumes after a Pod is deleted from the API server, i.e. the volumes are unmounted or someone force-deleted the pods. Very artificial reproducer: a Pod with an attachable CSI volume. the node, make the volume mount busy, e.g.: Every unmount of the volume fails and kubelet/the CSI driver periodically retry. the pod 6 minutes, see the volume is detached without unmounting first. KCM logs: Note that the Pod is still in the API server: And the node is healthy: Note that some CSI drivers (e.g. the cloud ones) do not allow detaching a volume while it's still mounted. Not all CSI driver have such possibility and they rely on Kubernetes not to call ControllerUnpublish before NodeUnpublish / NodeUnstage succeeds. <detailsToday's master </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-longterm
Right now, a volume is detached from a node when: (all pods that use the volume are deleted from the API server OR all their container are stopped) AND (the volume is unmounted OR 6 minutes passed). Potential solution 1: Issue detach of a mounted volume only when all pods are force-deleted from the API server.

More formally: Detach when [(All pods that use the volume are deleted from the API server OR all their container are stopped) AND the volume is unmounted] OR [all pods that use the volume are deleted from the API server AND 6 minutes passed]. Is this change subtle enough that it does not break existing behavior (e.g. tooling on top of Kubernetes)?
Potential solution 2: Add a new flag to CSIDriver, say . We would need to extend internal Detach() call to include a flag if the volume to detach is mounted or not (A/D controller knows that) and the CSI volume plugin would check CSIDriver and potentially reject detach of a mounted volume. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 3: As a variant, we could add a cmdline option to the external-attacher, like . When that cmdline option is specified, the external attacher would check , if the volume to detach is still mounted, and refuse such detach (with a reasonable error message). CSI driver vendors could opt-in into this behavior. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 4: As a variant, we could add to CSIDriver, where each driver could put their own timeout before force-detaching a volume, with an option to use infinity.
Note to self: check , it might help on the CSI side.
Solution 3 will not work well with ungraceful node shutdown - attach/detach controller does not watch pods and force-deleting pods is the main point of ungraceful node shutdown KEP. /me will try if solution 1 would work when NodeStage times out (is Pod deleted from the API server before or after NodeUnstage is called?)
was right, a pod is deleted (without force) from the API server after NodeUnpublish succeeds, it does not wait for NodeUnstage to finish. Therefore we cannot distinguish if a pod was force-deleted (and we should start 6 minute timer) or pod was deleted without any force and only NodeUnstage is taking more time than expected. Solution 1 is then not possible.
Now we are down to solution 2 and 4.

For solution 2, if driver does not have a way to check whether volume is mounted or not, the behavior will not change and we still force detach after 6 mins. And it might happen in the middle of unstage ..
Summing up yesterday's call: A/D controller should check if a node is unhealthy before force detaching volumes from it. It should never force detach a volume from a healthy node, not even after 6 minutes. Question what healthy means is open and we should talk to sig-node. is a working assumption.
I agree with both of these. I'd just emphasize that that particular definition of unhealthy definitely includes perfectly healthy nodes in a network partition. This change is no worse than the status quo ante for those nodes, and a definite improvement on the healthy nodes, so I wholeheartedly support it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign


Question: package token import ( <ins> "errors" </ins> "fmt" "sync" "time"

/kind bug /priority failing-test /priority important-critical /milestone v1.11 /assign cc All GKE jobs are failing in various release dashboards. Code freeze for 1.11 is a couple of hours away.
Common error appears to be:
Failures seem to align with when was merged. But nothing in that PR seems suspicious. Having verify. Bringing up my own cluster on head to try and get a live cluster.
I have a quick try to reproduce one of the failure test cases by gcloud container clusters create --addons=HttpLoadBalancing,HorizontalPodAutoscaling,KubernetesDashboard --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --cluster-version=1.11.0-beta.1.222+ test- The cluster isn't up. What I observed is the following: 1) kubelet on master is in crash-loop due to a panic No other master components could be up. 2) kubelet is in crashloop in node as well.
/sig node Looks like has a patch up at
/priority critical-urgent
/status approved-for-milestone
Waiting for new runs to verify the fix. Will reopen this issue if needed
/reopen Reopening this one until its actually verified to ensure it shows up in our tracking queries.
I see the GKE tests cleaning up in the . still doesn't seem to have the change (will have to wait for cutover?). The fix looks promising. Will wait for a couple more runs before I close
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Action Required: This issue has not been updated since Jun 6. Please provide an update. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release.

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
all the GKE ones still look good /close


Question: Unstructured(). NamespaceParam(cmdNamespace).DefaultNamespace(). FilenameParam(enforceNamespace, &options.FilenameOptions). <ins> Local(). </ins> Flatten(). Do() if err := r.Err(); err != nil { return err } <ins> dl, err := NewDownloader(f) if err != nil { return err } </ins> err = r.Visit(func(info *resource.Info, err error) error { if err != nil { return err } <del> if err := info.Get(); err != nil { if !errors.IsNotFound(err) { return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving current configuration of:n%snfrom server for:", info.String()), info.Source, err) } info.Object = nil } </del> <ins> remote, _ := dl.Download(info) </ins> obj := InfoObject{ <ins> Remote:  remote, </ins> Info:    info, Parser:  parser, Encoder: cmdutil.InternalVersionJSONEncoder(),

/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.


Question: // update rules: add needed for _, expectedRule := range expectedRules { foundRule := false <del> if findRule(updatedRules, expectedRule) { </del> <ins> if findRule(updatedRules, expectedRule, wantLb) { </ins> klog.V(10).Infof("reconcileLoadBalancer for service (%s)(%t): lb rule(%s) - already exists", serviceName, wantLb, *expectedRule.Name) foundRule = true }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the Idle Timeout annotation was set to "3" which isn't a supported case, kube-controller-manager is stuck in deleting Azure LB. New service with the same name couldn't be created again even without the annotation. What you expected to happen: New service should be created successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug
/assign Since I'm seeing this on some of our clusters
/priority important-soon
Thanks
- The problem here seems that when we want to call (when processing a delete event) it reconciles the loadbalancer which in turn tries to get the idleTimeout, resulting in an error. one of the strategies is to still construct our diff in case of a delete and exercise the code path at the end. This basically translate into ignoring the error when getting idleTimeout and excluding it from finding if the rule is dirty or not when we are . toughts ? Generally speaking, whenever we update a loadbalancer with unsupported values on annotations, ensuring load balancer will fail resulting in an un-cleaned LB, this is a cross-provider issue. /sig cloud-provider cc
cc
lgtm. the delete operation should continue to do cleanup job even some wrong options are provided from users.


Question: "encoding/base32" "fmt" "net" <ins> "os" </ins> "reflect" "strconv" "strings"

continuing discussion from and
/sig network /triage accepted
from I think we'll officially declare that we support kernel 5.4+ and nftables command-line 0.9.2+, and we can use a expression to check that. (Which means really we'll support kernel 5.3, but we won't advertise it that way.) FTR (and maybe worth documenting in a comment wherever we put the check): If we decide we need concatenations-with-ranges support again later, that would imply 5.6/0.9.4. If we want to use in sets/maps, that would require at least 0.9.4 (and maybe later depending on exactly how we're using it; we'd have to re-check the release notes). Support for comments on tables/chains/sets/maps/etc (anything but rules and elements) didn't come until 5.10/0.9.7, but knftables itself already does a for that at construct time, and just drops those comments if they're not supported.
There was a and about nft segfaults. It appears that specifically, nft v0.9.4 to v0.9.8 (inclusive) will segfault if asked to do anything when any chain in any table contains a certain kind of set/map as created by nft v1.0.3 or later. Specifically, rules like these trigger it (I think specifically it's caused by using and together, which the anonymous map as-generated-by-1.0.3+ uses. Using a named map rather than an anonymous one does not help.) Note that with nft < 1.0.1, doing any operation requires parsing the entire current ruleset. So that means when kube-proxy is using nft 1.0.3 or newer, nothing else in the host network namespace can use nft 0.9.4 - 0.9.8 for anything_. if the system firewall or something else in the host network namespace is using nft 1.0.3 or newer, then it might create rules that will break kube-proxy if kube-proxy is using nft 0.9.4 - 0.9.8.

If kube-proxy is using nft 1.0.1 or later, then it should not be possible for any other well-behaved component to break it, even if there are future bugs like the current one, because kube-proxy never looks at any tables but its own, so it wouldn't see things that other components did that it couldn't understand. OK, so what does this mean for us? We could just ship the official kube-proxy images with hand-built nft 0.9.2 rather than whatever's current in Debian. Then we guarantee we don't install rules other people can't deal with. Except that then we potentially run into problems if the system uses a much newer nft with some new bug that breaks 0.9.2. (We could do something -esque here, but ... ugh.) I think we actually want to go the opposite way, and have kube-proxy print a warning at startup if your nft is older then 1.0.1, because older versions run the risk of being broken by something someone else is doing in another table. But then there's the problem of us breaking other people. Or, even if we don't break other software, we still make it that admins on some systems would be unable to debug nftables-based problems... I guess we could avoid using the known-problematic syntax... the simplest way to do this would be to revert back to doing endpoints the way that iptables does: of course, that's less efficient, but...
Sounds to me that 1.0.2 is the golden version. It doesn't break 0.9.8, and it doesn't parse the entire current ruleset.
golden relative to the bugs we know about right now anyway... but maybe building our official images with 1.0.2 for a while would be a good plan... (Unfortunately we would have to actually build it ourselves. We can't just pull a 1.0.2 build from a random older Debian release because Debian never shipped that version; bullseye had 0.9.8 and bookworm had 1.0.6.)
thoughts on "use less efficient rules" vs "build images with a very specific nft version"?

build images with nft 1.0.1 seems the best option to me , because it seems that nft < 1.0.1 is not prod ready based on the different behaviors we are observing. If this happens between minors in 1.x.y then I think is a different discussion
The question wasn't just about what version we should use in our images, it was how we should deal with the bad interactions with older versions of nftables. It's one thing to say "nftables kube-proxy will not work when using nft < 1.0.1", but it's another thing to say "you are not allowed to run nftables kube-proxy on hosts where anybody is using nft < 1.0.1 (because otherwise kube-proxy will break other users of nft)". There is no reliable way for us to figure out the version of nft on the host from within the kube-proxy container. The best we could do would be to assume that the nft and kernel versions are roughly contemporary, and use the kernel version as a proxy. nft 1.0.1 was contemporary with kernel 5.16, and nft 1.0.0 was contemporary with 5.13. If we require kernel 5.13 / nft 1.0.1, we can support distros newer than mid 2021: RHEL 9 (kernel 5.14 + backports / nft 1.0.4) Debian 12 / bookworm / newstable (kernel 6.1 / nft 1.0.6) Ubuntu 22.04LTS (kernel 5.15 or newer / nft 1.0.2) (Not sure about SLES?)

Recent Fedora and other non-LTS distros Whereas our current plan of requiring kernel 5.4 / nft 0.9.2 means "newer than late 2019" and adds: RHEL 8 (kernel 4.18 + many backports / originally nft 0.9.3, later 1.0.4) Debian 11 / bullseye / oldstable (kernel 5.10 / nft 0.9.8) Ubuntu 20.04LTS (kernel 5.4 or newer / nft 0.9.3) (Not sure about SLES?) OK, so first decision: do we feel like we need to support running nftables kube-proxy on RHEL 8 / Debian 11 / Ubuntu 20.04LTS, or are we OK with only supporting the newer distros? For comparison, seems to be leaning toward 4.18/4.19 (from late 2018) as the minimum supported kernel for kubernetes as a whole, so requiring kernel 5.13 for nftables would add several more years before we could really even consider making it the default backend for kube-proxy. (We could also have kube-proxy require the newer versions by default, but have a config option to allow the older versions, and document exactly what can go wrong with that.) If we do want to still support the older distros (either unconditionally or via config option), there are two more questions: we decouple our kernel and nft requirements, and say that kube-proxy requires nft = 1.0.1 regardless of what kernel you're using? (I think "yes"?) we want to do some sort of workaround for the segfault bug (either changing the endpoint rules, or shipping the kube-proxy image with nft == 1.0.2 rather than whatever-version-is-currently-in-debian)? (If we had a config option to allow older kernels, we could make that option also control whether we used vmap-style or iptables-style endpoint rules?)

OK, leaning toward: Docs: We document that nftables kube-proxy requires nftables = 1.0.1, and recommends a kernel = 5.13 We also document that, currently, it can be used on systems with kernels as old as 5.4 if you set the option in the kube-proxy config, but (a) this will likely cause problems with other nftables users on the system (with details), and (b) this option is intended to go away at some point as the LTS distros with older kernels become unsupported. Code: We add to We check the nft client version (via a new knftables API), and if it's < 1.0.1, we error out. We check the kernel version (via ) and if it's < 5.13: if is (the default) we error out if is : if an with a expression passes, we continue, after warning that we might break other users on the system if the fails, we error out because the kernel is too oldWe don't change our rules or change the version of nft in our images, because the problems will only affect users who have explicitly opted in.
oh, and CI apparently has a 5.15 kernel, so we won't need to update the kind job config
I envision we'll hit more of these things in the future, so it will be nice you implement it, in a modular way, kind of the iptables runner that has all these different behaviors hidden from the users
The idea was that for things that affect knftables's own functioning, or its public API, it would check those things itself and hide the details from the caller. But the caller would be expected to use to figure out things like "can I use in a rule", since knftables can't just abstract around that (in general). (Right now the only thing knftables abstracts around is whether table/chain/set/map comments are supported; you're always allowed to specify them in the API, but .) I was originally thinking that the "require nft 1.0.1" restriction would be at the kube-proxy level rather than the knftables level, but I'm thinking now it would make more sense to require that for all knftables users.


Question: usingEviction                   bool getPodFn                        func(string, string) (*corev1.Pod, error) onDoneFn                        func(pod *corev1.Pod, usingEviction bool) <ins> onFinishFn                      func(pod *corev1.Pod, usingEviction bool, err error) </ins> globalTimeout                   time.Duration skipWaitForDeleteTimeoutSeconds int out                             io.Writer

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog