Question: golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= <ins> golang.org/x/exp v0.0.0-20190731235908-ec7cb31e5a56/go.mod h1:JhuoJpWY28nO4Vef9tZUw9qufEGTyX1+7lmHxV5q5G4= </ins> golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY= golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=

For structured logging migration we have developed a tool to klog invocations For a given directory logcheck validates two things: directory is fully migrated - check if there are non-structured logging calls like usage of and is correct - check if key values passed are paired. We run logcheck as part of Kubernetes verify scripts on directories listed in Directories listed in are marked as migrated to avoid regressions from organized migration, however there are also other code locations that already use new functions that are not yet ready to be marked as migrated (components that are not part of organized migrations are done as best effort). We should split checking if directory was migrated from correctness of new functions and scan whole codebase for correctness. Plan: Add a flag to logcheck for requiring that directory was migrated. Add to verify-strucured- running logcheck of whole k/k codebase to check correctness This will ensure higher quality of migration PRs by validating correctness of directories that were not yet in .structuredlogging /wg structured-logging /cc
I'd like to pick this up. Do I need to wait till it is triage/accepted?
/triage accepted
Feel free to pick it up :P Usually I don't want to self triage, but this issue is purely Structured Logging WG related.
/assign
[Case 2: Correct usage] If the directory is not marked as migrated, we check that it's usage of is correct and don't care about usage of . By default, CI will test for correctness of whole k/k codebase instead of testing for complete migration (which is the current behavior). And to test for migration, a new flag must be passed explicitly. is this the expectation?
To clarify, let me rephrase. Current state is What we want: [Case 2: Correct usage] tested on whole repository
If this is the case, I can start implementing it.
SG, thanks for the help.
/milestone v1.22 Part of work go graduate Structured Logging KEP to beta.
I don't think work here has finished. We still need to implement those changes in k/k repo: Cut new klog Bump klog in k/k Update the script ./hack/verify-structured- /reopen
Reopened this issue. <detailsIn response to : I don't think work here has finished.

We still need to implement those changes in k/k repo: Update the script /reopen </details
Why do we need to cut a new klog and bump in k/k? Can't we make latest logcheck available to k/k without it? I don't know the process.
Hmm, We could try without updating to update. Please send PR that will update ./hack/verify-structured- If updating klog is nesesery feel free to contact for help
If we do both, we'd have to run logcheck twice (with and without flag) which means test time will increase.
This depends on implementation, increasing running time is ok as long it's reasonable, for example x1.5, but not x10.
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
We would like to have static analysis setup before we start migrating logs in 1.23. will you be able to work on it this week? Do you need any help?
Yes, I can. PR kubernetes/kubernetes/pull/ is up for enabling static analysis. It has some unresolved conversations. It'd be helpful if I could get another round of review on that PR. Before we merge that we'd need to fix existing logging issues which are listed here: If I could get help fixing those it'd be great. Else I can fix it too.
I see two issues that need to be addressed: Fix handling of in klog code, this should be done by creating an issue to klog () Fix remaining errors Upgrade klog version to pick up fix for Exit


Question: - mountPath: /var/lib/kube-proxy/kubeconfig name: kubeconfig readOnly: false <ins> - mountPath: /run/xtables.lock name: iptableslock readOnly: false </ins> volumes: - hostPath: path: /usr/share/ca-certificates

BUG REPORT Kubernetes version 2 Environment: Install tools: What happened: The code in takes care to supply the flag so it will wait on a lock to avoid interference between multiple processes modifying the table. If you read the source you discover it does this by creating a file . However, when is running in a container, is in a local filesystem, so it is not able to lock against processes running on the host or in other containers. And before you decide that the answer is to mount from the host, look at all the issues around leaking netns, e.g. Credit for pointing this out to me. Credit for pointing out locking functionality was to a few months back
Ironically, this is caused by a bugfix that moved to file-based locking. Previously, iptables would lock by reserving an abstract unix socket, which is tied to the network namespace.
Some discussion also in and following comments.
I experimented with this: but it didn't work on CoreOS, as the host wasn't using a version of iptables that used the file lock yet. Just ended up creating a directory call
I guess I should file this as an issue against kubeadm and any of the other sources of kube-proxy container config.
/assign
Should we leave this open (waiting for ... ?)
Waiting for I guess... (which I'm reviewing now)
i think this should not be marked as complete until the kubeadm fixes are in.
/reopen
/close fix in


Question: for name, vmEntry := range oldCache { // if the nil cache entry has existed for 15 minutes in the cache // then it should not be added back to the cache <del> if vmEntry.virtualMachine == nil || time.Since(vmEntry.lastUpdate) > 15*time.Minute { </del> <ins> if vmEntry.virtualMachine == nil && time.Since(vmEntry.lastUpdate) > 15*time.Minute { </ins> klog.V(5).Infof("ignoring expired entries from old cache for %s", name) continue }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: From the comment above it should be How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /area provider/azure /sig cloud-provider


Question: } vmID, err := pc.InstanceID(nodeName) <ins> if err == cloudprovider.InstanceNotFound { glog.Infof("Instance %q does not exist, its disks will be detached automatically.", nodeName) // make all the disks as detached. return attached, nil } </ins> if err != nil { glog.Errorf("Photon Cloud Provider: pc.InstanceID failed for DiskIsAttached. Error[%v]", err) return attached, err

Cloud provider volume has a function DiskIsAttached to check whether the volume is still attached to the instance. If the instance is no longer exist (deleted by cloud provider), we can safely assume that the volume is no longer attached to the instance and return false without error for DiskIsAttached(). Currently GCE PD and AWS has the correct behavior, but all other volume plugins do not.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
In order to be easier to review, I split it for every cloud provider.
cinder plugin (openstack, rackspace):
Anyone can merge this PR:?


Question: case err := <-injectedError: return nil, err } <del> // Never get here. return nil, nil </del> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When reviewing Windows kubelet logs noticed errors like: What you expected to happen: The stats should be returned How to reproduce it (as minimally and precisely as possible): This occurs when the pods has just started or during termination. To simulate the issue: Then deploy pod: Anything else we need to know?: The container is in a state but is has not started running so the call to get the stats fails: This is handled in other places like moby: and Environment: Kubernetes version (use ): 1.18+ Cloud provider or hardware configuration: aks-engine OS (e.g: ): Windows Kernel (e.g. ): 2019 Install tools: Network plugin and version (if this is a network-related bug): Others: Dockershim /sig windows /priority critical-urgent /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:


Question: } } <ins> // SetExternalTrafficPolicy sets the externalTrafficPolicy field for a Service. func SetExternalTrafficPolicy(policy api.ServiceExternalTrafficPolicyType) Tweak { return func(svc *api.Service) { svc.Spec.ExternalTrafficPolicy = policy } } </ins> // SetAllocateLoadBalancerNodePorts sets the allocate LB node port field. func SetAllocateLoadBalancerNodePorts(val bool) Tweak { return func(svc *api.Service) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: was closed as not a problem with kubectl, but with the server What you expected to happen: 'kubectl replace' with no change on a service should work. How to reproduce it (as minimally and precisely as possible): reproduction included in this issue Anything else we need to know?: Seems to me like this comment sums up the options I got here from this issue Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
After a long day of reading and experimenting with the Kubernetes API i found a work-around I want to share with you. Problem in my case: I'm using the Java Client K8 API, especially I got the exact same error like written in and here in this issue summary: I send an update of a service WITHOUT specifying the spec.clusterip within the V1Service object. The request gets then rejected with the reason . Workaround: First load the existing service that contains the current clusterIP. Set the old clusterIp to the updated V1Service. Also load the old metadata.resourceVersion value and set it to the new V1Service object, because leaving the value out using the Java API Client results in an error: Please do not close this issue due to inactivity again, like it was done with . Even if there are some workarounds, this issue will pop up again and again, because there really is a problem with the API restrictions or at least the messages you will get when interacting with the API. Constructive Suggestion: e.g. instead of throwing the error you could throw something like and for the resourceVersion:
Seriously, this has been going on for years. is over 2 years old and there have been dozens of bug reports/feature requests (in Kubernetes, Kubectl, Helm, and elsewhere) showing the problems the current behavior is causing. (List available on request.) is from over 3 years ago. is from 2015.

The old justifications for keeping the current behavior ("works as designed") have proven to be insufficient. It is time for a redesign. The API should accept as meaning "assign a clusterIP" on creation and "keep the current clusterIP" on update. Since, when you set on creation, the resource does not end up with , this means a resource cannot validly have , and since the field is immutable, could never be a correct value for modification under the current API. While this could be considered a breaking change, I doubt there is anyone who would complain about it. Rejecting as a modification is not helping anyone. It is not rejecting an accidental change or avoiding confusion because the meaning is unclear, and it is hard for me to imagine a use case where someone is relying on the current behavior and would be upset about the proposed new behavior (with the possible exception of people writing compliance tests). It is just causing problems for the sake of some kind of theoretical purity that is in the eye of the beholder. Treating as meaning "system should replace this with a valid setting" is equally pure.
cc long-term-issue (note to self)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
keepalive
Is there something I can do to move this forward?
/remove-sig cli /sig network /assign this is sort of the inverse of the issue fixed in Using with a static manifest on an object that has spec fields populated by the API server (like clusterIP in service) or controller (like replicas in deployment) is an anti-pattern. is a better match for this usage. Alternatively, a controller that did a 'read -modify -replace' could be ok, since it would be starting with the current API server state.
This is not impossible to fix in the PUT path (), but it basically turns PUT into PATCH. We have distinct PATCH operations for a reason, and that is almost certainly what you want in this case. should "just work" as a replacement for . Is there a reason why you can't use PATCH or apply?

- while I continue to argue that is wrong here, maybe we should just cave and make it quietly patch the value of clusterIP in this specific case? I really dislike it, not just for purity reasons, but because I have a hard time seeing if that will bite us later. Thoughts?
I really dislike it as well. "replace all spec content" is a bad thing to teach clients to do on resources that have fields set by different actors.
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. - My inclination is to close this. I know it's not what you want, but I do think you and most users will be better served by patch or apply in almost all cases (even knowing that Service has an ambiguity around changing ports that overlap each other which is not easily fixed because it is breaking). Is there a reason why you can't use patch or apply?
This really seems related to bug in Helm, as far as what I have read, and some recent (Jan2021) posts suggest that was fixed. Have not tested this out myself. Agree that adhering to expected PUT/PATCH api behavior would be best. Thanks for the followup!
I reject the idea that this is a bug in Helm, even though Helm is working hard (and failing) to handle the situation. I agree with you when you "Agree that adhering to expected PUT/PATCH api behavior would be best". My point is that the expected "PUT" behavior is that if a "PUT" request succeeds, the exact same "PUT" request given a second time should succeed as well. That is what is broken and I am asking to have fixed. Is there a better reason why I should be prohibited from using replace on a Service, when I can use it for every other resource? Why should Service require special-case handling by the user? I ask you to think of this in the larger context of using Infrastructure as Code (IaC) to manage resources, and the automation around that. In the IaC definition of a Service, the has to be set to in order to get a system-assigned address. Semantically, says "make the resource exactly like this". It can be, and is, used to ensure there is not "drift" in the configuration, i.e.

that the configuration stored in the IaC code base is live. It is the correct thing to use when you want to say "here is the configuration that should be active", and it is what automation uses to reset the configuration after it has drifted. Users should not have to treat Services specially, they should be able to any resources with the same spec that created them. Stating this another way, (A user should be able to the same thing repeatedly, successfully, and without causing changes after the first accepted .) By not accepting for in , you are breaking the contract of making idempotent. (As a matter of practicality, the Helm maintainers are trying to make up for the fact that is not idempotent, but that is not an excuse for Kubernetes to leave this API in its current state.) As I stated above, I am not even advocating for this case to be handled as a . As I see it, the value of is handled specially but not idempotently. The value of currently means one thing if the resource does not exist and something else if it does exist. I am asking instead that it consistently, idempotently be taken as meaning "any valid address: the existing one if one is set, otherwise one randomly assigned by the controller".
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. Unfortunately, to make PUT idempotent in this case, it would need to turn into a variant of PATCH. I'm not even saying I totally disagree, but that is what it means. As for PUT being idempotent, I don't think that's ever really been a hard objective. In this case (and look, I am not saying I am happy about it) that means reading existing state and patching it into the PUT. We can do that (probably) but I can't say for sure that there are no other APIs where this will happen (there probably are not). Note that to do this we would need to fixup , , and probably the newer dual-stack-related fields. Can someone help me understand the "bug"? Is it about using PUT instead of apply? I could see two paths to fixing this, if we want to (and I am still not 100%). 1) Change clusterIP and nodePort fields to pointers.

If not-specified on PUT AND the object exists AND the old warrants a value AND the new warrants a value, copy the existing values. This is a bit tricky for nodePort we'd have to at least specify what happens if the list is reordered or otherwise modified. For example, if the existing object has: and the update specifies: Do I track the move by name? What if the names changed, does that lose the tracking? We've had innumerable issues here because the API-specified merge-key was (is) ambiguous. The only way to get exactly what you want in ALL cases is to RMW the resource. Explicitly setting "" and 0 would work on initialization (compat) but would not be auto-patched. 2) Leave the fields as non-pointers and accept "" and 0 as "DWIM" mode. I dislike this because it makes it less obvious when a user mistakenly tries to clear a field - it used to fail and will now "succeed" but silently change the value. I am not sure I see a clever 3rd option. The only reason I entertain this at all is because of the issues around apply/patch and ports. NOTE: My willingness to think it thru is not a commitment to implement it. I am swamped right now :)
wrote I am shocked that you would think that, although I don't really know what you mean by "hard objective". The doc certainly implies should be idempotent, via such statements as and Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. But that stuff should already be handled by the more generalized resource handling architecture.

The general rule about PUT is that PUT on an existing resource should adjust that resource to be in the same working configuration as if you used that PUT to create a new resource. Calling PUT a second time with the same input should be acceptable and should not change the resource. I believe that Kubernetes generally has a canonical ordering of all fields in a resource (and if there is no clear way to order a list, the canonical order is simply the order in which the list was originally specified on resource creation), so what you do is enumerate the fields of the existing resource in canonical order, filter out the fields like that PUT cannot specify, and then compare it to the PUT resource, after normalizing it into canonical order, and add/delete/replace field by field. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . All we are asking for is that this case be interpreted differently and allowed. is documented as an allowable value in the input, but it is never an allowable value in the realized resource, so it cannot possibly mean to be interpreted as "force this value on the resource". Its meaning is that the system should assign a valid value and the user does not care what it is. So on PUT, when is matched against the exiting , he API should just confirm that is a valid value and, if so, accept the input.
As for PUT being idempotent, I don't think that's ever really been a hard objective. What I meant is it's just never been a primary question in reviewing APIs. Service is a weird case, far outside the norm and I'm not excusing it - just explaining what I meant. Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. For historical reasons, the "key" for ports was not the name but the port value, then the value + protocol. So we can key off that, I think.

I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. Those are all outside of . I don't know of any resource that PUT-PATCH inside the spec. Hilariously, I just noticed that nodePort is even worse - it accepts the non-value and then re-allocates the nodePort. I am going to call this a bug, but now we have a compatibility discussion to have. If a user does not specify a value, we can reasoinably argue that we should retain existing values. But if they specify , do we stay compatible and re-allocate the node port, or do we say that was always a bug? Is it a useful "feature" to be able to re-allocate? We disallow it explicitly gfor ClusterIPs. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. You haven't spent much time in the Service REST code, I guess. I have a large rework of that on-going, but this was not part of it. :) The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . Given the previous point about nodePort, would it be fair to modify your statement as: "when PUT says (or does not specify at all) ..." With the corollart that if you actually specify you will get the current (immutable error) behavior. This seems like it would satisfy helm and other users who are not specifying literal "" - they are not specifying any value at all. Thoughts?
At this point I am willing to call it a bug, albeit an unpleasant one.
wrote I have had regular run-ins with this issue for years, which is why I'm being insistent on getting it fixed properly. I do not recall having similar problems with and so I am less informed on those issues. I don't know what means in the spec or in practice, so I don't have a specific opinion as to what it should do to an existing resource.

I will simply restate the general principle which should apply to just like everything else: what ever you allow as a setting on creation should be allowed in any and should not cause a change if repeatedly applied. The very practical reason for this is so that I can record some place (like a checked into a VCS) that "this is how the resource should be configured" and use that (1) to create the resource and (2) to return the resource to expected configuration after some external operation changed it, using a standard API while otherwise treating the resource as opaque.
/cc
/cc (i know i am going to regret this)
It is a bug. Not because it does not match rest api semantics. But because it is not consistent with the rest of the apis. In other words, If user can do on other resources they should be able to do it on as well. It is hard (and weird) to ask users to do one thing for all resources, but then do a totally different thing for Services. As discussed over sig-network the problem will be user intent. Some ideas for a solution: If irrespective of changes in other we can assume that the user is replacing and does not care about ClusterIP value, hence copying to new should be fine. It is basically like the user saying so copying old=new should be Ok. If that is a failure case, because it breaks ClusterIP immutability rules ( we should really, really remove ClusterIP immutability but i digress). We will still require the validation to work as is. e.g. replacing with with a type that does not require ClusterIP and not setting it to should fail. I can look into building a POC for this.
I am probably missing some subtlety here, but as in I suspect that Helm is behaving properly and the problem is in way Kubernetes handles s: the of a resource should never be rewritten with variable environmental information; that is what is for. In other words could be set in the unusual case that you wished to pick an IP yourself but would normally be null; would reflect the service’s current value, expected to be retained across unrelated patches.
nailed it on the head with his last comment. I do agree that we should not be re-defining the behaviour of a PUT request.

But the inconsistency of how a Service updates the field compared to other resources results in inconsistencies in the PUT behaviour. Once a resource defines fields in the object, normally Kubernetes leaves it unchanged. My understanding is that Kubernetes uses this field to use as a reference point to be the "ideal state of the world", using the field to record the "current state of the resource". The only offending case I've seen in the wild is a , as it updates both the field AND the field. The end result is that a with a defining results in differing behaviour than the rest of the resource APIs. From Helm's point of view, a chart author's only course of action is to a) omit from the manifest and all subsequent updates, or b) if you happened to define in your first install, find the value Kubernetes provided and inject it into your template so that the value Kubernetes provided is preseved. Failing to do so results in a failed PUT () request. I wrote this behaviour down in-depth in this Helm comment: I'd propose an alternative solution than solutions: instead of updating after an IP has been assigned to the Service, use the API, i.e. . It would be a breaking change to the API, but it should bring Services back in line with the rest of the resource APIs, leaving as a user-defined field.
I agree
I'm afraid that breaking an API is not a feasible option
That document does allow for breaking changes under the apiVersion. Instead of making a breaking change to v1 services, you can introduce Service v2. Existing v1 services migrate their spec.clusterIP’s existing value (auto-generated or not) to the new version. spec.clusterIP and status.clusterIP would now match their corresponding relationships, and Kubernetes retains backwards/forwards compatibility.
(If it's not clear whether the API changes document does or doesn't allow the change from last comment, let's raise a separate improvement request against that document).
Does the case is important ? Because here I can read or ...
I have a patch. I'll try to get it out for review tonight.
Has anyone yet checked that the purported fix actually addresses the issue in Helm?
Please do!
I can still see the error but I'm on GKE 1.18, will this only apply to Kubernetes 1.21/22?

It will be part of 1.22 or the next 1.21.X (I don't remember now what is the latest stable)
Can anyone confirm that this issue was actually fixed in k8s version 1.21.4 or 1.22.1 ? I do not see it listed in the release notes/change log
The change was released in the 1.22.0-beta.2, so yes it should be available in 1.22. See I do not believe it was backported to 1.21.
Thanks for confirming and providing the details


Question: return addrs[0].Address, nil } <ins> //getStringFromServiceAnnotation searches a given v1.Service for a specific annotationKey and either returns the annotation's value or a specified defaultSetting func getStringFromServiceAnnotation(service *v1.Service, annotationKey string, defaultSetting string) string { glog.V(4).Infof("getStringFromServiceAnnotation(%v, %v, %v)", service, annotationKey, defaultSetting) if annotationValue, ok := service.Annotations[annotationKey]; ok { //if there is an annotation for this setting, set the "setting" var to it // annotationValue can be empty, it is working as designed // it makes possible for instance provisioning loadbalancer without floatingip glog.V(4).Infof("Found a Service Annotation: %v = %v", annotationKey, annotationValue) return annotationValue } //if there is no annotation, set "settings" var to the value from cloud config glog.V(4).Infof("Could not find a Service Annotation; falling back on cloud-config setting: %v = %v", annotationKey, defaultSetting) return defaultSetting } </ins> // TODO: This code currently ignores 'region' and always creates a // loadbalancer in only the current OpenStack region.  We should take // a list of regions (from config) and query/create loadbalancers in

Is this a BUG REPORT or FEATURE REQUEST?: Feature request /kind feature What happened: Currently we have kubernetes cluster installed, in node machines we have one subnet and two routers attached. One router for external network and one router for internal network. However, when we create type: loadbalancer services in kubernetes, we have to hardcode floatingnetwork id to only one of those (external or internal). What you expected to happen: I except that I could define from which floatingnetwork loadbalancer is taking its floating ip address in each service. One possible solution could be that if floatingnetworkid is defined like currently - that networkid is default one. Then we could have possibility to override that in each service configuration. Like if we do not want use default floating-pool we could change it: I do not know does this even work in openstack, it might need some manual route settings in lbaas. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: openstack OS (e.g. from /etc/os-release): centos cc
Exposing first to external network so the external floatingip is working. Then update floating-networkid in kubernetes cloud-config in all nodes+masters and restart all machines. Exposing to internal network However, this last curl did not work from outside openstack. There is some route issue with it, if I try that outside openstack the result is: tcpdump from node the external one which works 192.168.1.12 is lbaas fixed ip: tcpdump from node the internal one which does not work 192.168.1.13 is lbaas fixed ip: This might mean that lbaas is using default router (which is external) to answer back to client, so the route in lbaas should be modified. No idea is that possible.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig network
I have defined host route to subnet, which is 10.222.0.0/16,192.168.1.4 this row basically means that all connections toward 10.222.0.0/16 should use 192.168.1.4 router interface, which is internal router. However, openstack lbaas is ignoring these host routes so before that is fixed there is no sense to code this feature to kubernetes. edit. looks like it is fixed that fix is not to our openstack yet. After that fix this is this my idea should work. Now the question is how we should implement this multiple floatingnetworkids to kubernetes. I can make PR but first I would like to discuss how it should be done.
Sounds good. cc
the problem what I am thinking currently is that I would like to have possibility to specify floating-pool in each service. Some service people may want expose service to external network and some service to internal network. However, the service api spec is quite hardcoded or can I add variable there? I am planning to add variable like there.


Question: t.Fatal(spew.Sdump(test.Infos)) } } <ins> func TestFlattenListVisitorWithVisitorError(t *testing.T) { b := newDefaultBuilder(). FilenameParam(false, &FilenameOptions{Recursive: false, Filenames: []string{"../../artifacts/deeply-nested.yaml"}}). Flatten() test := &testVisitor{InjectErr: errors.New("visitor error")} err := b.Do().Visit(test.Handle) if err == nil || !strings.Contains(err.Error(), "visitor error") { t.Fatal(err) } if len(test.Infos) != 6 { t.Fatal(spew.Sdump(test.Infos)) } } </ins>

What happened: When using a yaml file for input, silently fails at the first missing resource, not returning any further results. What you expected to happen: Searching with cli arguments properly ignores missing resources and continues the query, eg: Would expect similar results with a file input, eg: How to reproduce it (as minimally and precisely as possible): With an input file of : the command fails at apparently the first missing resource: Note: that rearranging the inputs in (so that the missing resource is last) produces the "expected" output: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: local OS (e.g: ): fedora 30 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: x-ref:
/sig cli
/assign
Got around to reproducing it and can confirm that I'm getting the same behavior. I'm looking into fixing it now
thanks, any updates on this?
I've learned that in the case of a fake entry, kubectl stops sending API get requests altogether. Learned by running kubectl get -f --ignore-not-found --v=10, the result is shown below. Till now I've been looking into the cobra command framework but now I'm trying to find out where kubectl is making these API calls from(working on finding out where this connection is happening from Slack+debugging, I've asked a more detailed question here ). <img width="1440" alt="Screen Shot 2019-08-12 at 4 13 41 PM" src="https://user-"
that's an interesting find, definitely helpful. When I looked into this I got to the section of get's main run function, and it seems like it should be continuing on any errors (see ) but maybe the ResourceBuilder is actually stopping when it hits a 404?
This behavior comes from where the not found errors are being ignored and where the input list is flattened. Errors are ignored here: While flattening occurs here: As the visitor iterates through the objects in the list if an error occurs, such as not found, it will stop iterating over the objects in the list and return that not found error. The logic will then ignore it and continue.

Any objects that came before the not found object in the list will have been visited and will then be included in the output. To make the behavior consistent between the two ways of getting the set of objects, the would need to be updated to support filtering errors.
Just submitted a PR adding that functionality to the FlattenListVisitor. The behavior is now consistent between the two methods.


Question: return false } <del> klog.Infof("OpenAPI AggregationController: Processing item %s", key) </del> <ins> if aggregator.IsLocalAPIService(key.(string)) { // for local delegation targets that are aggregated once per second, log at // higher level to avoid flooding the log klog.V(5).Infof("OpenAPI AggregationController: Processing item %s", key) } else { klog.Infof("OpenAPI AggregationController: Processing item %s", key) } </ins> action, err := c.syncHandler(key.(string)) if err == nil {

What happened: Set up an APIService object pointing to a service that does not service openapi What you expected to happen: Aggregation would attempt to aggregate, fail, and back off How to reproduce it (as minimally and precisely as possible): openapi aggregation logs continuously /sig api-machinery /cc
actually, this is happening without any external APIService objects registered as well
This is just too verbose logging. We have nothing better than polling of the delegation chain for changes. But the good thing is that with etags this should not be heavy on the CPU because nothing is recomputed (and if it is, it would be a bug).
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
Kube-apiserver outputs the same log every second and takes up 10% more CPU than normal
cc
This problem has been fixed in some . Someone has covered up the problem. It makes it harder for us to find the CPU problem behind it. thanks.


Question: make test-integration WHAT=./test/integration/scheduler_perf KUBE_TIMEOUT="-timeout=3600s" KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-alsologtostderr=false -logtostderr=false -run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling -cpuprofile ~/cpu-profile.out" ``` <ins> ### How to configure bechmark tests Configuration file located under config/performance-config.yaml contains a list of templates. Each template allows to set: - node manifest - manifests for initial and testing pod - number of nodes, number of initial and testing pods - templates for PVs and PVCs - feature gates See `simpleTestCases` data type implementation for available configuration for each template. Initial pods create a state of a cluster before the scheduler performance measurement can begin. Testing pods are then subject to performance measurement. The configuration file under config/performance-config.yaml contains a default list of templates to cover various scenarios. In case you want to add your own, you can extend the list with new templates. It's also possible to extend `simpleTestCases` data type, respectively its underlying data types to extend configuration of possible test cases. </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/test/component/scheduler/perf/README.md?pixel)]() <!-- END MUNGE: GENERATED_ANALYTICS -->

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We are making a series of enhancements to scheduler benchmarks. Once we are done, we should update the Ref , Why is this needed: /sig scheduling cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: SyncPeriod:    metav1.Duration{Duration: 5 * time.Second}, MinSyncPeriod: metav1.Duration{Duration: 2 * time.Second}, }, <del> Mode: kubeproxyconfig.ProxyModeIPVS, </del> <ins> Mode: proxyMode, </ins> IPVS: kubeproxyconfig.KubeProxyIPVSConfiguration{ SyncPeriod:    metav1.Duration{Duration: 10 * time.Second}, MinSyncPeriod: metav1.Duration{Duration: 5 * time.Second},

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network Fix windows UT for proxy mode. Run UTs for in windows, That's because proxy mode "IPVS" is not valid in windows. What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign


Question: function start_nodelocaldns { cp "${KUBE_ROOT}/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml" nodelocaldns.yaml <del> # .* because of the __PILLLAR__ references that eventually will be removed ${SED} -i -e "s/_.*_DNS__DOMAIN__/${DNS_DOMAIN}/g" nodelocaldns.yaml ${SED} -i -e "s/_.*_DNS__SERVER__/${DNS_SERVER_IP}/g" nodelocaldns.yaml ${SED} -i -e "s/_.*_LOCAL__DNS__/${LOCAL_DNS_IP}/g" nodelocaldns.yaml </del> <ins> # eventually all the __PILLAR__ stuff will be gone, but theyre still in nodelocaldns for backward compat. ${SED} -i -e "s/__PILLAR__DNS__DOMAIN__/${DNS_DOMAIN}/g" nodelocaldns.yaml ${SED} -i -e "s/__PILLAR__DNS__SERVER__/${DNS_SERVER_IP}/g" nodelocaldns.yaml ${SED} -i -e "s/__PILLAR__LOCAL__DNS__/${LOCAL_DNS_IP}/g" nodelocaldns.yaml </ins> # use kubectl to create nodelocaldns addon ${KUBECTL} --kubeconfig="${CERT_DIR}/admin.kubeconfig" --namespace=kube-system create -f nodelocaldns.yaml echo "NodeLocalDNS addon successfully deployed."

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gci-gce-serial-kube-dns-nodecache Which test(s) are failing: N/A Since when has it been failing: 10/13 Testgrid link: https://k8s- Reason for failure: Cluster failed to start: Anything else we need to know: /cc
So, is a failing job, we can see that there is a error around binding to a specific IP address. In passing jobs, the initial core file isn't printed out () so not sure how to do an apples to apples comparison of what is broken here yet.
/kind bug I have been looking at this offline with and The only difference between a failing and a passing job is the arguments to the bind plugin. Failing Job: Link - Image - Config: Passing Job: Link - Image: Config:
Issue is reproducible when spinning up a cluster via kubetest using:
changes to lines 907-910 in hack/local-up- and lines 2467 to 2469 in configure- seem like the issue.
What we see is substitution works correctly for line , but for line , it replaces "PILLARLOCALDNS" with empty string.
/assign Can you investigate?
Yeah lookin into it
Regex is matching the entire line :). Should be easy fix . Will verify in hack local up and issue PR shortly
/sig network /triage accepted


Question: if err != nil { t.Fatalf("Couldn't create cacher: %v", err) } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.ResilientWatchCacheInitialization) { // The tests assume that Get/GetList/Watch calls shouldn't fail. // However, 429 error can now be returned if watchcache is under initialization. // To avoid rewriting all tests, we wait for watchcache to initialize. if err := cacher.Wait(context.Background()); err != nil { t.Fatal(err) } } </ins> d := destroyFunc s = cacher destroyFunc = func() {

pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.


Question: c.indices[name] = index } <ins> if len(indexValues) == 1 && len(oldIndexValues) == 1 && indexValues[0] == oldIndexValues[0] { // We optimize for the most common case where indexFunc returns a single value which has not been changed continue } </ins> for _, value := range oldIndexValues { <del> // We optimize for the most common case where indexFunc returns a single value. if len(indexValues) == 1 && value == indexValues[0] { continue } </del> c.deleteKeyFromIndex(key, value, index) } for _, value := range indexValues { <del> // We optimize for the most common case where indexFunc returns a single value. if len(oldIndexValues) == 1 && value == oldIndexValues[0] { continue } </del> c.addKeyToIndex(key, value, index) } }

This was the root cause of rancher/rancher. In , when is called, it is given the oldVersion of an object and the newVersion of the object. It then uses the indexers to calculate the old values (from the old version) and the new updated values. In cases where are size of 1, are of size greater than 1, and contain the sole value from , the key will be deleted for the old value will be deleted and won't be re-. This is due to the two for loops at the end of the method which are used for optimization. When the first for loop is processed, . Because of this, the key is deleted from the index for this value. In the second for loop, the , and for at least one of the values . The continue statement causes the key to not be to the index for this value (and since the value was deleted in the prior for loop, it no longer exists in the index for this value), despite the fact that the indicated that this is the desired state. Since specified that this value was still valid for this object, the key should not be deleted from the index for this value, or it should be re- when it is re-discovered in . Either way, the end state of the index should include this key in the set for the specified value. I've created a which demonstrates the issue (branch is ). To use it (assuming that you have a running kubernetes cluster to use with): the controller: . the foo crd: the controller (assumes that you have $KUBECONFIG set to the file path of your kubeconfig file). an example foo with only one label: . Observe, in the logs of the running sample-controller, that the indexer was able to locate this foo using the indexer. the foo to include a second label: . Observe, in the logs of the running sample-controller, that the indexer was not able to locate the foo using the first key any longer, despite the fact that the label was still on the object. The sample-controller fork adds an indexer for foo objects which uses the indexerFunction of . This function looks for labels that have a key with a prefix of and uses their values as index keys for the foo objects. For example, if I have a with a label of , the indexer will allow you to retrieve using .

The controller's method has been altered to just ensure that the foos can be retrieved using the indexer for all valid labels. If the controller can't retrieve foo by using the value from a valid label, then it returns an error (it tries for all valid labels). This was originally discovered in a fork of client-go that we maintain. After we reverted the changes introduced by , our issues stopped occurring. N/A - This was introduced in and appears to be in all versions of the client v0.23+. N/A - issue is with client-go No response No response No response No response
/sig api-machinery
I think that you have background in this issue/area of the code. Would you mind taking a look at this issue and the PR that I have up to fix it? Please let me know if I've misunderstood anything.
/assign Would you mind to take a look? Thank you /triage accepted
Thanks for reporting the issue. will address this issue
Reopening - since we need to cherrypick back to 1.23
Cherrypick is merged - closing.


Question: klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s)", nodeResourceGroup, vmName, diskName) ctx, cancel := getContextWithCancel() defer cancel() <del> if _, err := as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM); err != nil { </del> <ins> _, err = as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM) if err != nil { </ins> klog.Errorf("azureDisk - detach disk(%s) failed, err: %v", diskName, err) } else { klog.V(2).Infof("azureDisk - detach disk(%s) succeeded", diskName)

What happened: After running a rolling-update on a StatefulSet that uses Azure Disks, part of the replicas remain in a state forever, exhibiting the following events: Restarting the occasionally unblocks the situation: What you expected to happen: Attach/detach operations succeed consistently. How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brAnd the default StorageClass: <details</summary</details</brExtra information: Two other kind of Pod events that are regularly associated with the error above: kubelet logs of a node where attaching the volume is failing kube-controller-manager logs in an environment where attaching the volume is failing Environment: Kubernetes version (use ): v1.10.10 Cloud provider or hardware configuration: Azure (westeurope) OS (e.g. from /etc/os-release): 16.04.4 LTS Kernel (e.g. ): 4.13.0-1018-azure Install tools: ACS-Engine Others: /kind bug /sig storage /sig azure this might be of interest for you, since you were involved in , which was supposed to fix attach/detach issues in Azure.
/assign
fix the attach disk issue in kubelet, while in your case, you restarted the controller manager, and then it works. It looks like disk volume is never attached to vm successfully before you restart controller manager.
Indeed. For completeness I updated the description and 2 more type of events that are regularly associated to this issue.
could you check activity logs in that resource group, the time should be before you restart controller manager, you could check whether there is any disk related issue. Since you have already restarted controller manager, we cannot get all logs before your restart.
I see a bunch of the following failures in the activity log, but the errors stopped 4h before I gathered the logs and restarted the controller-manager: <details<summaryactivity log</summary</details</brI managed to grab the kube-controller-manager logs from an environment where the situation is currently happening. See the updated description.
Hi would you add my slack: andyzhang workspace?

I think your issue is here, you are using strategy According to , if statefulset replica is using azure disk pvc, is not the right way, pls change the strategy to 'Recreate' Update: may be not related to that issue, still in investigation.
In our deployment every Pod uses a distinct Azure Disk, so there is not multi-attachment. I'm still waiting for feedback from customers, hoping this happens again today.
I have tried in my testing env with v1.10.10 on vmss nodes by , there could be error like following after disk attached to that VM, the disk detect action in vmss node seems slow sometimes, while in the end it could be recovered:
thanks for looking into it. In our case the disk never gets attached (1d in ContainerCreating state).
Hi I found an regression issue of , will send a PR soon. Thanks for reporting.


Question: "iface-127.0.0.1:3260:pv1/127.0.0.1:3260-iqn.2003-01.io.k8s:e2e.volume-1-lun-3", "iface-127.0.0.1:3260:pv2/127.0.0.1:3260-iqn.2003-01.io.k8s:e2e.volume-1-lun-2", "iface-127.0.0.1:3260:pv2/192.168.0.1:3260-iqn.2003-01.io.k8s:e2e.volume-1-lun-1", <ins> filepath.Join(config.DefaultKubeletVolumeDevicesDirName, "iface-127.0.0.1:3260/192.168.0.2:3260-iqn.2003-01.io.k8s:e2e.volume-1-lun-4"), filepath.Join(config.DefaultKubeletVolumeDevicesDirName, "iface-127.0.0.1:3260/192.168.0.3:3260-iqn.2003-01.io.k8s:e2e.volume-1-lun-5"), </ins> } for _, d := range subdirs {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deletion of a Pod with an ISCSI volume in mode results in an error during , leaving an ISCSI target logged into on the kubelet node despite the target being unused. In my case, the ISCSI Target server enforces a single session per LUN, which blocks the Pod from starting on another Node in the cluster. What you expected to happen: For the kubelet to issue a logout request for the given target once the refcounter hits zero. How to reproduce it (as minimally and precisely as possible): Have an ISCSI Target server Create a target/LUN that contains an ext4 filesystem. Have a Node configured as an ISCSI Initiator capable of logging into the those targets. Create a Pod on the Node that includes an volume referring the LUN, using (rather than ) mode. minimally: Delete the Pod Wait for the Pod to be destroyed and the related ISCSI block device to be removed. on the Node will show the target is still logged in Kubelet error: Anything else we need to know?: Environment: Kubernetes version (use ): 1.16.2 Cloud provider or hardware configuration: Reproducible on a bare metal (Intel NUC) lab and DigitalOcean DOKS. OS (e.g: ): Ubuntu 18.04 Kernel (e.g. ): 4.15 (NUC) / 4.19 (DigitalOcean) Install tools: Ubuntu package /sig storage


Question: ctx, cancel := getContextWithCancel() defer cancel() klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s)", nodeResourceGroup, nodeName, diskName) <del> if _, err := ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm); err != nil { </del> <ins> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm) if err != nil { </ins> detail := err.Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) { // if lease cannot be acquired or disk not found, immediately detach the disk and return the original error

What happened: After running a rolling-update on a StatefulSet that uses Azure Disks, part of the replicas remain in a state forever, exhibiting the following events: Restarting the occasionally unblocks the situation: What you expected to happen: Attach/detach operations succeed consistently. How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brAnd the default StorageClass: <details</summary</details</brExtra information: Two other kind of Pod events that are regularly associated with the error above: kubelet logs of a node where attaching the volume is failing kube-controller-manager logs in an environment where attaching the volume is failing Environment: Kubernetes version (use ): v1.10.10 Cloud provider or hardware configuration: Azure (westeurope) OS (e.g. from /etc/os-release): 16.04.4 LTS Kernel (e.g. ): 4.13.0-1018-azure Install tools: ACS-Engine Others: /kind bug /sig storage /sig azure this might be of interest for you, since you were involved in , which was supposed to fix attach/detach issues in Azure.
/assign
fix the attach disk issue in kubelet, while in your case, you restarted the controller manager, and then it works. It looks like disk volume is never attached to vm successfully before you restart controller manager.
Indeed. For completeness I updated the description and 2 more type of events that are regularly associated to this issue.
could you check activity logs in that resource group, the time should be before you restart controller manager, you could check whether there is any disk related issue. Since you have already restarted controller manager, we cannot get all logs before your restart.
I see a bunch of the following failures in the activity log, but the errors stopped 4h before I gathered the logs and restarted the controller-manager: <details<summaryactivity log</summary</details</brI managed to grab the kube-controller-manager logs from an environment where the situation is currently happening. See the updated description.
Hi would you add my slack: andyzhang workspace?

I think your issue is here, you are using strategy According to , if statefulset replica is using azure disk pvc, is not the right way, pls change the strategy to 'Recreate' Update: may be not related to that issue, still in investigation.
In our deployment every Pod uses a distinct Azure Disk, so there is not multi-attachment. I'm still waiting for feedback from customers, hoping this happens again today.
I have tried in my testing env with v1.10.10 on vmss nodes by , there could be error like following after disk attached to that VM, the disk detect action in vmss node seems slow sometimes, while in the end it could be recovered:
thanks for looking into it. In our case the disk never gets attached (1d in ContainerCreating state).
Hi I found an regression issue of , will send a PR soon. Thanks for reporting.


Question: shouldDeny:     true, }, } <del> for _, c := range evictionCases { </del> <ins> for i := range evictionCases { c := evictionCases[i] </ins> expectation := "should allow an eviction" if c.shouldDeny { expectation = "should not allow an eviction"

Due to a bug in the disruption e2e test, we really only run one test case 5 times instead of all 5 test cases. This will be fixed when is merged.


Question: srcs = [ "controller.go", "doc.go", <del> "patch.go", </del> ], importpath = "k8s.io/kubernetes/pkg/controller/service", visibility = ["//visibility:public"], deps = [ "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", <del> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", <del> "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/util/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <del> "//staging/src/k8s.io/apimachinery/pkg/util/strategicpatch:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", "//staging/src/k8s.io/client-go/informers/core/v1:go_default_library",

PR service patch methods to Once there is a release with that PR included, the code in service-controller and legacy-cloud-providers/gce can be updated to use those. /area platform/gce /assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Created a PR to use these new functions in Service-controllers.


Question: // same status. newStatus.ObservedGeneration = rc.Generation <del> var getErr error </del> <ins> var getErr, updateErr error var updatedRC *v1.ReplicationController </ins> for i, rc := 0, &rc; ; i++ { <del> glog.V(4).Infof(fmt.Sprintf("Updating replica count for rc: %s/%s, ", rc.Namespace, rc.Name) + </del> <ins> glog.V(4).Infof(fmt.Sprintf("Updating status for rc: %s/%s, ", rc.Namespace, rc.Name) + </ins> fmt.Sprintf("replicas %d->%d (need %d), ", rc.Status.Replicas, newStatus.Replicas, *(rc.Spec.Replicas)) + fmt.Sprintf("fullyLabeledReplicas %d->%d, ", rc.Status.FullyLabeledReplicas, newStatus.FullyLabeledReplicas) + fmt.Sprintf("readyReplicas %d->%d, ", rc.Status.ReadyReplicas, newStatus.ReadyReplicas) +

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: framework.ExpectNoErrorWithOffset(0, err) for _, node := range nodeList.Items { if node.Labels[gkeNodepoolNameKey] == poolName { <ins> node := node </ins> nodes = append(nodes, &node) } }

At , within the loop , variable is shared by all the loop iterations, as described in the Go specification (). All instances of &pv within the loop may refer to the last tem updated by the loop. As a consequence, may hold multiple instances of the same item. Later on these remaining PVs are deleted, so it is possible that the same is deleted multiple times: More information on the bug can be found at A fix is to avoid reusing the shared variable node: Found by #deepgo.
/triage accepted /area e2e-test-framework
Thanks for pointing out, makes complete sense. I'm happy to fix it. /assign
/sig testing

At , within the loop , variable is shared by all the loop iterations, as described in the Go specification (). All instances of &node within the loop may refer to the last node item updated by the loop. As a consequence, may hold multiple instances of the same node. Note that the following unit test checks only the number of nodes rather than the nodes themselves (otherwise the bug may be revealed): More information on the bug can be found at A fixe is to avoid reusing the shared variable : Found by #deepgo.
/triage accepted /area e2e-test-framework /sig testing
Thanks for pointing out, makes complete sense. I'm happy to fix it. /assign


Question: depends on container runtime and preconfigured handler. Runtime-specific functionality is not being tested here. */ <del> ginkgo.It("should schedule a Pod requesting a RuntimeClass without PodOverhead [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should schedule a Pod requesting a RuntimeClass without PodOverhead [NodeConformance]", func() { </ins> rcName := createRuntimeClass(f, "preconfigured-handler", e2enode.PreconfiguredRuntimeClassHandler, nil) defer deleteRuntimeClass(f, rcName) pod := f.PodClient().Create(e2enode.NewRuntimeClassPod(rcName))

Per the comment in this PR conformance tests will regress because of a newly Runtime Class GA v1 API is not covered with tests. Full comment: Full comment: /assign More links:
/sig node /triage accepted
KEP:
New to conformance tests. So the idea is to mark tests (or create new ones) that would call each of these methods at lease once? So if this test will be marked as conformance, it will cover and . Is there any additional requirements for the conformance tests? Maybe with the number of side effects of that test or a special execution environment?
Yes. You can make a single test that runs objects through CRUD operations. For the deletecollection one, I'd suggest labeling the objects under test and scoping the deletecollection to your test label to avoid breaking the cluster you're running on. See as an example
Great, will be working on it! Also just chatted with about it.
Just to get clarity on this - is this considered a blocker for Confirming bc grad criteria for it says "No release blocking feedback for API and functionality" should these tests be required to consider this stable? As a note: Code Freeze means that required tests must also be finished:
Here's possibly an easier link to track: !
This seems like something that should be tested generically, or at the very least have a table-driven test that covers all (or most) GA APIs.
! Just a quick note to show that currently the new tests are hitting all endpoints. I'd go ahead and create the Promotion PR in order to get any necessary feedback from #sig-arch
/reopen re: I disagree with the idea that a single test covering CRUD operations means RuntimeClass is now conformance tested. Generic CRUD on a resource's API is not sufficient to describe the intended behavior of the resource. That kind of test is at best verifying that api machinery is working. A vendor would able to pass this test, without implementing any of the behaviors runtimeclass enables. What does do? What kinds of behaviors are visible to an end-user? Are any of those tests in Conformance?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I'm not familiar enough with runtimeclass to know whether these would be a good start:
Thank you for reopening. I agree with this in principle. But I really would appreciate some guidance here. Scheduling tests that takes runtime handler into account are great. However they don't hit any endpoints. I asked and I'm not clear whether those needs to be covered as they are not touching an API. As for actual scheduling, one limitation is posted . Which makes some other tests impossible to include to conformance. I next week, but there may not be all the right people there as it's a Thanksgiving short week. Do you have a suggestion?
I agree that exercising specific CRI implementations is not the place of Conformance testing. Conformance testing is intended to exercise end-user visible behaviors of a Kubernetes cluster, which basically means whatever you can see and do at the Control Plane API level (ref: ) The challenge we face with Conformance testing is that we don't have an easy way to enumerate and measure coverage of behaviors. We experimented with listing behaviors in .yaml so that humans and machines could populate and approve them independent of test implementation, but ultimately abandoned that effort due to its complexity (ref: ) So we use API coverage as an easily machine-measurable/enforceable proxy. Noticing the coverage was at 0% should have been a sign that we should look at the existing test coverage of RuntimeClass and decide which tests were eligible for promotion to Conformance, and whether any new tests needed to be written. In theory a solid test plan for a KEP would include a list of behaviors that need to be tested. That's not immediately clear to me from the KEP for RuntimeClass (ref: ) Based on my read of the KEP, some valid tests might be: tests that exercise the runtimeclass admission controller creating pods with a non-existent runtimeclass creating pods with an existing runtimeclass creating pods with scheduling rules that would clash with runtimeclass scheduling rules creating pods with scheduling rules that would merge with runtimeclass scheduling rules Tests that would be disallowed by conformance requirements because they would require pre-installation/configuration of the cluster under test: running pods with a runtimeclass (unless there's some way to create a functional "dummy" runtimeclass without touching kubelets) running pods with a runtimeclass with a specific handler such as containerd, crio, etc.

and expecting them to run
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
sorry for the delay /remove-lifecycle stale
/area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: message HTTPIngressPath { // Path is matched against the path of an incoming request. Currently it can // contain characters disallowed from the conventional "path" part of a URL <del> // as defined by RFC 3986. Paths must begin with a '/'. When unspecified, // all paths from incoming requests are matched. </del> <ins> // as defined by RFC 3986. Paths must begin with a '/' and must be present // when using PathType with value "Exact" or "Prefix". </ins> // +optional optional string path = 1;

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I tried to use the default path I got . No error. According to the path is not needed. "When unspecified, all paths from incoming requests are matched." Create a ingress like so: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: brew Network plugin and version (if this is a network-related bug): N/A Others:
/sig network /area ingress It seems to be a problem of ingress and I suggest creating an issue in I find a similar one which is a little different there
We use - would this be a problem with the aws repo then?
/assign
So digging a bit here, the API docs needs to be improved. By the , is required if the PathPrefix is Exact or Prefix. To solve your issue specifically, you have two ways: Add that accepts ingress without paths Add and change PathPrefix to Prefix, so this becomes a catchAll I'll open a PR to fix the comment/docs. Thanks!
Thanks! In my case because I'm using ALB controller I had to use with prefix to get it working - see
/triage accepted PR will close this ;)


Question: "-j", "DROP", ) <ins> // disable LB access from node // for IPVS src and dst both would be lbIP for _, entry := range proxier.lbNoNodeAccessIPPortProtocolEntries { proxier.filterRules.Write( "-A", string(kubeIPVSOutFilterChain), "-s", entry.IP, "-m", "ipvs", "--vaddr", entry.IP, "--vproto", entry.Protocol, "--vport", strconv.Itoa(entry.Port), "-j", "DROP", ) } </ins> // Accept all traffic with destination of ipvs virtual service, in case other iptables rules // block the traffic, that may result in ipvs rules invalid. // Those rules must be in the end of KUBE-SERVICE chain

When using loadBalancerSourceRanges to allow access only from CIDRs outside the cluster segment, access from the K8s nodes (or pods with hostNetwork:true) should be rejected, but for proxy-mode=ipvs it works. /sig network /area kube-proxy /area ipvs Access from K8s nodes should be rejected in the same way as for proxy-mode=iptables Define a service with loadBalancerSourceRanges that allows access from a cidr outside the cluster, e.g. The K8s nodes have addresses on 192.168.1.0/24 and fd00::c0a8:100/120. Make sure a loadBalancerIP is assigned (in this example 10.0.0.0/32) and try to access it from a K8s node: Same for both ipv4 and ipv6. This is related to since a fix requires that the passed to the is correct. Please see v1.27.4 and master None Linux 6.4.4 None N/A N/A
wanted to work on this
Great! But please coordinate with I hope that one get accepted, and then ~the s (both families) passed to the proxiers should be usable.~ No, they are not, please see
Discussed this one in the SIG Network meeting today: we agree this is something we'd like to implement, however we weren't sure exactly how to do it with ipvs during the meeting. Whoever takes this on needs to come up with and communicate a clear plan of how it will be accomplished. /triage accepted
My comment is not feasible. For proxy-mode=ipvs the src and dst is both loadBalancerIP. So the nodeIP will not help :disappointed:
I did a small experimented with kind + metallb. I created a loadbalancer service backed by . loadBalancerIP: 172.18.255.200 podIP: 10.244.1.3 conntrack picks the src and dst both as loadBalancerIP. Tried to create an iptables rules matching source and destination with loadbalancerIP to drop packets but it didn't work. After some exploration I found and with the following rules I was able to prevent access to lbIP from k8s nodes / host network.

The solution looks hacky though, we only need this rule if source ranges are defined for the service and node ip is not covered by any of them. Will try to program this in the ipvs proxier.
Good find! :+1: I had missed the ipvs extension. I started to check nodeIP and the (flawed) option to experiment with setting a nodeIP as "src" for some routes to dodge the lbip-both-as-src-and-dst problem, and an old "vision" I have to not assign clusterIPs to at all, but have one rule to replace them all, like: It can be done, and in a cluster with 1000's of clusterIP's it would simplify


Question: permReason2    = "Permanent2" permMessage2   = "permanent error 2" ) <del> var source, config, tmpLogFile string </del> <ins> var source, config, hostLogFile string </ins> var lookback time.Duration var eventListOptions metav1.ListOptions

Leads to this avc: In general, HostPath volumes in tests are bad news for selinux.


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package priorities import ( "reflect" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" nodeinfosnapshot "k8s.io/kubernetes/pkg/scheduler/nodeinfo/snapshot" ) func TestNodePreferAvoidPriority(t *testing.T) { annotations1 := map[string]string{ v1.PreferAvoidPodsAnnotationKey: ` { "preferAvoidPods": [ { "podSignature": { "podController": { "apiVersion": "v1", "kind": "ReplicationController", "name": "foo", "uid": "abcdef123456", "controller": true } }, "reason": "some reason", "message": "some message" } ] }`, } annotations2 := map[string]string{ v1.PreferAvoidPodsAnnotationKey: ` { "preferAvoidPods": [ { "podSignature": { "podController": { "apiVersion": "v1", "kind": "ReplicaSet", "name": "foo", "uid": "qwert12345", "controller": true } }, "reason": "some reason", "message": "some message" } ] }`, } testNodes := []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{Name: "machine1", Annotations: annotations1}, }, { ObjectMeta: metav1.ObjectMeta{Name: "machine2", Annotations: annotations2}, }, { ObjectMeta: metav1.ObjectMeta{Name: "machine3"}, }, } trueVar := true tests := []struct { pod          *v1.Pod nodes        []*v1.Node expectedList framework.NodeScoreList name         string }{ { pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "default", OwnerReferences: []metav1.OwnerReference{ {Kind: "ReplicationController", Name: "foo", UID: "abcdef123456", Controller: &trueVar}, }, }, }, nodes:        testNodes, expectedList: []framework.NodeScore{{Name: "machine1", Score: 0}, {Name: "machine2", Score: framework.MaxNodeScore}, {Name: "machine3", Score: framework.MaxNodeScore}}, name:         "pod managed by ReplicationController should avoid a node, this node get lowest priority score", }, { pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "default", OwnerReferences: []metav1.OwnerReference{ {Kind: "RandomController", Name: "foo", UID: "abcdef123456", Controller: &trueVar}, }, }, }, nodes:        testNodes, expectedList: []framework.NodeScore{{Name: "machine1", Score: framework.MaxNodeScore}, {Name: "machine2", Score: framework.MaxNodeScore}, {Name: "machine3", Score: framework.MaxNodeScore}}, name:         "ownership by random controller should be ignored", }, { pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "default", OwnerReferences: []metav1.OwnerReference{ {Kind: "ReplicationController", Name: "foo", UID: "abcdef123456"}, }, }, }, nodes:        testNodes, expectedList: []framework.NodeScore{{Name: "machine1", Score: framework.MaxNodeScore}, {Name: "machine2", Score: framework.MaxNodeScore}, {Name: "machine3", Score: framework.MaxNodeScore}}, name:         "owner without Controller field set should be ignored", }, { pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "default", OwnerReferences: []metav1.OwnerReference{ {Kind: "ReplicaSet", Name: "foo", UID: "qwert12345", Controller: &trueVar}, }, }, }, nodes:        testNodes, expectedList: []framework.NodeScore{{Name: "machine1", Score: framework.MaxNodeScore}, {Name: "machine2", Score: 0}, {Name: "machine3", Score: framework.MaxNodeScore}}, name:         "pod managed by ReplicaSet should avoid a node, this node get lowest priority score", }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { snapshot := nodeinfosnapshot.NewSnapshot(nodeinfosnapshot.CreateNodeInfoMap(nil, test.nodes)) list, err := runMapReducePriority(CalculateNodePreferAvoidPodsPriorityMap, nil, nil, test.pod, snapshot, test.nodes) if err != nil { t.Errorf("unexpected error: %v", err) } // sort the two lists to avoid failures on account of different ordering sortNodeScoreList(test.expectedList) sortNodeScoreList(list) if !reflect.DeepEqual(test.expectedList, list) { t.Errorf("expected %#v, got %#v", test.expectedList, list) } }) } } </del>

/sig scheduling /priority important-soon See as an example. Part of This shouldn't need a PostFilter Plugin, it is not worth caching the result of metav1.GetControllerOf. /assign


Question: "//test/utils/image:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library", "//vendor/github.com/onsi/gomega:go_default_library", <ins> "//vendor/k8s.io/utils/net:go_default_library", </ins> "//vendor/k8s.io/utils/pointer:go_default_library", ], )

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-network-kind-ipvs,dual,master sig-network-kind-ipvs,IPv6,master Which test(s) are failing: [x] KubeProxy should set TCP CLOSE_WAIT timeout [x] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort Service [x] KubeProxy should resorve connection reset issue (tracking in ) Since when has it been failing: Oct 28 Testgrid link: sig-network-kind-ipvs,dual,master  Reason for failure: Both appear to be failing on missing conntrack entries ex. Anything else we need to know:
/sig network /cc /assign
I'm spinning up a kind cluster with ipv6+ipvs to see if I can replicate this test locally
/triage accepted
heh, the job was not running it, it started to run when we changed the tag I meant this test and the PR where we renamed the kube-proxy test
this one has an issue already open, and I have a PR for fixing it
reached out to me on Slack and we identified as the high priority. I have tested this locally and it fails on both IPTables and IPVS kind clusters locally. I'm digging through the cluster logs now.
I' ve logging to conntrack with level v4 recently just for these problems, try to bump your kube-proxy logging to v4
seems that there are less test failures on the IPv6 jobs, let's keep watching them :eyes: However, the dual-stack jobs are legit failures, basically because on dualstack clusters, the node has 2 internal IPs, and those tests use the internal IP for asserting some operations, and the problem is that they are picking the internalIP of the wrong IP family, hence the test fails The solution should be to add the InernalIp correspoding to the cluster primary IP familty, you can obtain it with but it seems that we have to modify the function or add a new one that returns the nodes and the corresponding ips
Hey Thanks for all the help, will look into the areas of the code and try to bump the logging.


Question: if (pattern.VolMode == v1.PersistentVolumeBlock && t.runTestBlock != nil) || (pattern.VolMode == v1.PersistentVolumeFilesystem && t.runTestFile != nil) { ginkgo.It(t.testItStmt, func() { <ins> if pattern.VolMode == v1.PersistentVolumeBlock && driver.GetDriverInfo().InTreePluginName == "kubernetes.io/local-volume" { // TODO: https://github.com/kubernetes/kubernetes/issues/74552 framework.Skipf("Local volume volume plugin does not support block volume reconstruction (#74552)") } </ins> init() defer cleanup()

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (block volmode)] disruptive[Disruptive] Should test that pv used in a pod that is force deleted while the kubelet is down cleans up when the kubelet returns. Since when has it been failing: 10/8/19 Testgrid link: https://k8s- Reason for failure: Introduced by but root cause is probably Anything else we need to know:
/assign
/milestone 1.17 /priority important-soon this is consistently failing in release-master-informing job could you please take a look?
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.17 /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17


Question: }, { "ImportPath": "github.com/google/cadvisor/api", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/collector", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/container", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/events", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/fs", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/healthz", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/http", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v1", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v2", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/metrics", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/storage", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/summary", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/validate", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" </ins> }, { "ImportPath": "github.com/google/cadvisor/version", <del> "Comment": "0.13.0-22-ge3aa15a", "Rev": "e3aa15a3bbee5367532ac56aa55be41441c55356" }, { "ImportPath": "github.com/google/go-github/github", "Rev": "930e6fdb8dc2b11458fdeb55b3cd68e5370a1a28" }, { "ImportPath": "github.com/google/go-querystring/query", "Rev": "547ef5ac979778feb2f760cdb5f4eae1a2207b86" }, </del> <ins> "Comment": "0.13.0-32-g832c6e9", "Rev": "832c6e94c325ecdf05da884880c503be45594818" }, { "ImportPath": "github.com/google/go-github/github", "Rev": "930e6fdb8dc2b11458fdeb55b3cd68e5370a1a28" }, { "ImportPath": "github.com/google/go-querystring/query", "Rev": "547ef5ac979778feb2f760cdb5f4eae1a2207b86" }, </ins> { "ImportPath": "github.com/google/gofuzz", "Rev": "bbcb9da2d746f8bdbd6a936686a0a6067ada0ec5"

I have some reports of a Node reporting a a resource capacity for cpu as a negative value. Any idea why cpu could report as a negative value? Is this an expected normal outcome? In its current state, it prevents any pods from being scheduled to that Node. Tips on how to debug further are appreciated. - any ideas?
Can you check machineInfo on Cadvisor or kubelet on the node?
a.k.a:
It is a bug in cAdvisor side. I am assigning it to you :-)
here is contents of /proc/cpuinfo Pertinent pieces of a log:
Machine is AWS instance if that matters....
We build the topology from system/cpu info. I think we can return partial topology information and ignore caches if its not present. This isn't a recent change though. Is it happening on just a single machine? I'll file a cAdvisor issue.
Can you link the referenced issue once created?
Re-assigning to since he's working on google/cadvisor
Could be this was caused by getting a Node on EC2 with old hypervisor, and I am the first victim...
That sounds likely. Most of the cache information is there in /proc/cpuinfo. So looks like the sysfs bits are disabled.
Can you try this cadvisor image on your machine and check /api/v1.3/machine output: sudo docker run  --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw  --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080  --detach=true  --name=cadvisor  jnagal/cadvisor:canary
hmm
ah, selinux :( You'll need to run it with --privileged, I think. who's been dealing with it.
No luck with --privileged either on this box...
This has to do with SELinux, I think we had to disable it for this to work (some of our mounts are being denied). We're working on a better solution... This is not a problem in Kubernetes since we don't run Kubelet/cAdvisor in a Docker container.
ok, unfortunately, i have managed to kill docker on the box in question, and i cannot get the daemon to restart right now so I am not sure when that box will be available to debug against again.

I can provide an environment with a similar issue if required.
The fix is checked in to head. Can you build the binary and try?


Question: }, }) if err != nil { <del> return admission.NewForbidden(a, err) </del> <ins> return errors.NewInternalError(err) </ins> } // refuse to operate on non-existent namespaces

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: import ( "fmt" <ins> "sort" </ins> "github.com/go-logr/logr" json "k8s.io/component-base/logs/json"

What happened: 50% of builds have a different man page content What you expected to happen: It should be possible to get bit-identical build results - might need a patch similar to - knows these things. How to reproduce it (as minimally and precisely as possible): build kubernetes1.19 openSUSE package a few times and compare results. e.g. with Anything else we need to know?: Environment: OS: openSUSE-Tumbleweed While working on for , I found that when building the kubernetes1.19 package, there was sometimes such a difference between builds:
/sig release
/sig scheduling


Question: import ( "fmt" <ins> "sort" </ins> "sync" "time"

I've noticed a issue where if you have multiple replicated slave pods that don't have a HOST assigned and you reduce the number of replicas in the replicationController definition for those pods, sometimes the currently assigned (and running) pods get killed and the unassigned pod will get assigned to run in the killed pods place. The issue can be reproduced using the guestbook example provided in the repo. Steps to reproduce: (Note this is not always deterministic and may need multiple attempts) the master pod the master service the replicated slaves the Slave Controller definition to only have 1 replicas the Slave Controller with new definition The expected behavior should be that the scheduler is able to determine that unassigned pods are more viable candidates when reducing number of replicas. It seems to be more likely that since this issue is not deterministic, the ordering in which the command returns result is not always deterministically in the same order
: please can you look at this or suggested someone else? Thank you.
Unassign myself from this bug since I cannot get working on this soon. Also remove kubelet from the label since it is not kubelet issue. This is obviously a replicaController issue. cc/ Instead of destroying pod in pending state, replicaController unbound running pod from a node. Kubelet running on such node has this new desire state and kill unwanted pod to match it.
Yes, the replication controller should be smarter, taking into account Phase, Readiness, recency, ... This will be a significant problem for auto-scalers, but I expect manual scaling to be infrequent. I doubt we'll get to this soon. A PR would be welcome. Place to change is here:
Hi, I am final year Master's student of International Institute of Information Technology, Hyderabad. I am familiar with docker and containers. I have basic understanding of Go language with some hands on Docker API. This project seems interesting and a solution to this will also be a good addition to Kubernetes. So I would like to pick this as my GSoC project. I have started reading about Replication Controller and Pods and going though the code. I would appreciate if someone has any inputs for me on this. Thanks, Swapnil Patil


Question: "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" watch "k8s.io/apimachinery/pkg/watch"

[X] Test approval issue: [X] Test PR: 29 Sept 2022 [X] Two weeks soak end date: 14 Oct 2022 [X] Test promotion PR: According to this APIsnoop query, there are still two remaining ReplicationControllerScale endpoints which are untested for conformance. readCoreV1NamespacedReplicationControllerScalereplaceCoreV1NamespacedReplicationControllerScale Using a number of existing e2e test practices a new has been created for two ReplicationControllerScale endpoints. The e2e logs for this test are listed below. This query shows which scale endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 2 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Sometimes the actual containers take a second to get started, try to get logs for 60s for time.Now().Sub(start) < (60 * time.Second) { <ins> err = nil </ins> logs, err = c.Get(). Prefix("proxy"). Resource("nodes").

This happened on 5/6/2015 at 07:49 am on our continuous integration system, against upstream/master head at that time (i.e. last commit ). Needs to be repro'd and debugged. Docker Containers should be able to override the image's default command and arguments Failed to create pod: Pod "client-containers--f401-11e4-9118-" is forbidden: Post http://127.0.0.1:8080/api/v1beta3/namespaces: EOF
This test is still flaky. Containers should be able to override the image's default commmand (docker entrypoint) (Failed 2 times in the last 30 runs. Stability: 93 % Containers should be able to override the image's default arguments (docker cmd) (Failed 1 times in the last 30 runs. Stability: 96 %)
I will take a look at this now
Sample log from failure:
The indicates two things 1) that the test only tried once to fetch the logs, and 2) that either the call returned an empty log or an error (which it looks like the test does not check for). See