Question: } } <del> var strictDecodingErr error </del> <ins> var strictDecodingErrs []error </ins> obj, gvk, err := c.decoder.Decode(data, defaultGVK, decodeInto) if err != nil { <del> if obj != nil && runtime.IsStrictDecodingError(err) { // save the strictDecodingError and the caller decide what to do with it strictDecodingErr = err </del> <ins> if strictErr, ok := runtime.AsStrictDecodingError(err); obj != nil && ok { // save the strictDecodingError and let the caller decide what to do with it strictDecodingErrs = append(strictDecodingErrs, strictErr.Errors()...) </ins> } else { return nil, gvk, err } } <del> // TODO: look into strict handling of nested object decoding </del> if d, ok := obj.(runtime.NestedObjectDecoder); ok { if err := d.DecodeNestedObjects(runtime.WithoutVersionDecoder{c.decoder}); err != nil { <del> return nil, gvk, err </del> <ins> if strictErr, ok := runtime.AsStrictDecodingError(err); ok { // save the strictDecodingError let and the caller decide what to do with it strictDecodingErrs = append(strictDecodingErrs, strictErr.Errors()...) } else { return nil, gvk, err } </ins> } } <ins> // aggregate the strict decoding errors into one var strictDecodingErr error if len(strictDecodingErrs) > 0 { strictDecodingErr = runtime.NewStrictDecodingError(strictDecodingErrs) } </ins> // if we specify a target, use generic conversion. if into != nil { // perform defaulting if requested

With the addition of server side validation in 1.23, it is now possible for a call to successfully decode it’s data and return a non-nil error when there are strict decoding errors (e.g. duplicate/unknown fields). This means that callers of can distinguish between fatal errors (that cause decoding to fail) and strict decoding errors (that still allow the decode to succeed) by inspecting the target object (to see if it was successfully decoded into) and the error itself (to see if it is a ) With the NestedObjectDecoder however, it only returns an error from giving no indication whether a strict decoding error it has returned comes from an internal decode step that short circuited (by not accounting for strict decoding errors) or actually did successfully decode, but also encountered strict decoding errors. We propose taking a few actions to minimize the possibility of erroneous short-circuiting from Document how all nested decoders should only return strict decoding errors if all decoding is successful or return only non-strict decoding (ie. fatal errors). Document this for all known in-tree and out-of-tree . Update in-tree nested decoders to do the above mentioned behavior (which is just the kube-scheduler) Long-term solutions to better guarantee that no erroneous short-circuiting is taking place could include deprecating the current interface in favor of one that returns both the target object that was decoded into along with an error return a special error type specific to indicating that all internal decoding was successful.
/sig api-machinery
/triage accepted


Question: ginkgo.By("creating a TCP service " + serviceName + " with type=ClusterIP in namespace " + ns) jig := e2eservice.NewTestJig(cs, ns, serviceName) <ins> jig.ExternalIPs = true </ins> servicePort := 8080 tcpService, err := jig.CreateTCPServiceWithPort(nil, int32(servicePort)) framework.ExpectNoError(err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We ran sonobuoy on one of our clusters running on AWS and it reported two failed tests: This happened because the pod employed to conduct the tests wasn't able to reach the service using the node's public IP address: What you expected to happen: The security groups on that cluster don't allow for connections to arbitrary ports through the public IP address. The tests should still succeed because that configuration is seen as a security feature. How to reproduce it (as minimally and precisely as possible): Spin up a Kubernetes cluster on AWS EC2 instances, configure SGs to only allow traffic to the instances from other instances (or from an ELB). Run sonobuoy. Anything else we need to know?: Environment: AWS Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area conformance /sig network
related:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am also seeing this issue when running conformance testing. The nodes in our test cluster (where we run conformance tests) have public IP addresses, but nearly all ports all firewalled. We are also running in AWS. These tests should only rely on the internal IP address of the nodes.
/assign
Yes, I have seen this before, that if the nodes have an external address it will use that one, which means the firewall must be open for that. If the nodes do not have an external IP then the test will pass just fine. It's not as much a bug as an inconvenient implementation. But we can treat it as a bug and fix it. The intent of the test is to make sure the service is plumbed properly.
/remove-triage unresolved
Good point. However, we only assign an external IP address to be able to SSH into the host and thus block all other ports.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm seeing the same issue, though I noticed that it happens only on one node in the whole cluster. Even after re-creating a cluster from scratch, the test always fails only on one specific node. However this node is identical to others, but the test succeed on rest of the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
In some user scenarios, the external ip nodeport is for access from APPs outside this cluster/zone. However, in most scenarios, external IP is just for SSH or special use(Ingress Gateway, etc.) Different users have different proposals. Why not disable e2e tests checking on external IP with a flag? Will send a PR for this.
That's not all that useful for those running conformance tests. Would be better to change it to always use the internal IP, we don't need the external access for conformance purposes.
/assign
agree, also most of the network tests now run from the cluster, no need to reach externalIPs please tag me in the PR
So kubernetes applications should not expect to be able to access them with an external node address? FWIW we also require upstream / public internet access in the DNS tests, so needing somewhat unrestricted networking to test the cluster behavior is not new.
I'm not extremely familiar with the conformance tests. Are the DNS tests you are referring to about being able to access public services from inside of the cluster (egress)? Because our problem is that our default Kubernetes deployment doesn't allow ingress other than SSH and we haven't seen any problems with the DNS tests so far when checking our product for conformance. That's why I like approach of disabling that specific test with a flag.

Maybe that could also be generalized to let all conformance tests know this is a locked-up cluster with basically no ingress. But that may be a matter of a subsequent PR/issue.
Disabling the tests with a flag is not really an acceptable approach for conformance. The whole point of the conformance program is to validate the Kubernetes implementation against a standard set of tests across all vendors etc. If we don't think access like this is a standard Kubernetes behavior applications should expect anymore, the tests should be removed from conformance and we can consider instead running them just to test the Kubernetes project but not in conformance by moving them to a new test. IMHO we usually only increase the scope of conformance behavior though, not decrease, and it's expected that some users will have more secure clusters that fail conformance (for example we test some privileged behaviors IIRC) ...
After reading comment, I agree that the test should just be removed. However, I'd like to first understand the reasoning behind it being there in the first place. Maybe there's a good reason that such "secured cluster" aren't deemed compliant?!
so, the next step is to review the conformance definition for nodeport and modify the test accordingly Node.Status.Addresses are not a great thing, bad semantics, ... this KEP and this PR summaries perfectly why I personally, would move ExternalIP from Conformance entirely until it has a clear definition
So we should remove it now and then add a new test if needed after a conclusion from the discussion in the KEP. Edited. I may misunderstand. A proposal is to move those external addr NodePort test from conformance to normal e2e test cases In Conformance Case, don't check external ip's NodePort. In e2e test case, check them.
My point is that Conformance folks (cc ) should decide if they want to consider ExternalIPs as part of the conformance definition, I pointed to some issues related to the topic here sig-network will keep testing externalIPs, are part of the API and we need to verify them, Conformance has a narrower scope
The fact that these tests are using ExternalIPs does not need to be in conformance and in fact really shouldn't be. But we still need the tests working against InternalIPs for conformance. It's more of an historical artifact that they use ExternalIPs than an intentional choice.

So, agree with that sig-network should test them, but we should remove the use of the external IPs from conformance, and leave in the tests but against the internal ips only. I resolved this, for example, in GKE conformance runs by using a private GKE cluster that does not give the nodes external IPs.


Question: // createServiceReportErr creates a Service and reports any associated error. func createServiceReportErr(cs clientset.Interface, ns string, service *v1.Service) *v1.Service { svc, err := cs.CoreV1().Services(ns).Create(context.TODO(), service, metav1.CreateOptions{}) <del> framework.ExpectNoError(err) </del> <ins> framework.ExpectNoError(err, "error deleting Service") </ins> return svc }

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ## Changelog since v1.13.8 <del> **No notable changes for this release** </del> <ins> * Fix CVE-2019-11249: Incomplete fixes for CVE-2019-1002101 and CVE-2019-11246, kubectl cp potential directory traversal ([#80436](https://github.com/kubernetes/kubernetes/pull/80436)) * Fix CVE-2019-11247: API server allows access to custom resources via wrong scope ([#80750](https://github.com/kubernetes/kubernetes/pull/80750)) See also the [security announcement for this release](https://groups.google.com/forum/#!topic/kubernetes-security-announce/vUtEcSEY6SM). </ins> # v1.13.8

What would you like to be : The release notes state "No notable changes for this release": However it turns out these releases fix important security problems: Why is this needed: If people track releases in, for example, GitHub and then use the release notes to see what's changed, they will totally miss that important security problems have been fixed in these releases. People should not have to look in multiple places to find news of such fixes. The current practice risks people not upgrading because there were "no notable changes" and being oblivious to the risks they run.
I expect this is one for and probably others but I don't really have any clue who.
/sig release This has to do with the timing of cutting the releases vs sending the announcements. The notices were included in the announcement, but not in the generated md files. /assign can you open a PR to update those?
Looks like they put it in the email and the group, but not the release notes. !topic/kubernetes-security-announce/vUtEcSEY6SM/discussion
Just opened a PR to fix this: (I only noticed that this issue was already assigned after committing my changes, hope that's OK)
Thanks that PR looks exactly like what I'm after for this particular update!


Question: var csiTestDrivers = []func() testsuites.TestDriver{ drivers.InitHostPathCSIDriver, drivers.InitGcePDCSIDriver, <del> drivers.InitGcePDExternalCSIDriver, </del> drivers.InitHostPathV0CSIDriver, // Don't run tests with mock driver (drivers.InitMockCSIDriver), it does not provide persistent storage. }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Remove out of tree plugin config introduced in once we can refactor the e2e tests enough to be able to vendor them as a library in external repos according to /assign <!-- DO NOT EDIT BELOW THIS LINE --/king bug
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: oldService := old.(*api.Service) newService.Status = oldService.Status <ins> patchAllocatedValues(newService, oldService) </ins> NormalizeClusterIPs(oldService, newService) dropServiceDisabledFields(newService, oldService) dropTypeDependentFields(newService, oldService)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: was closed as not a problem with kubectl, but with the server What you expected to happen: 'kubectl replace' with no change on a service should work. How to reproduce it (as minimally and precisely as possible): reproduction included in this issue Anything else we need to know?: Seems to me like this comment sums up the options I got here from this issue Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
After a long day of reading and experimenting with the Kubernetes API i found a work-around I want to share with you. Problem in my case: I'm using the Java Client K8 API, especially I got the exact same error like written in and here in this issue summary: I send an update of a service WITHOUT specifying the spec.clusterip within the V1Service object. The request gets then rejected with the reason . Workaround: First load the existing service that contains the current clusterIP. Set the old clusterIp to the updated V1Service. Also load the old metadata.resourceVersion value and set it to the new V1Service object, because leaving the value out using the Java API Client results in an error: Please do not close this issue due to inactivity again, like it was done with . Even if there are some workarounds, this issue will pop up again and again, because there really is a problem with the API restrictions or at least the messages you will get when interacting with the API. Constructive Suggestion: e.g. instead of throwing the error you could throw something like and for the resourceVersion:
Seriously, this has been going on for years. is over 2 years old and there have been dozens of bug reports/feature requests (in Kubernetes, Kubectl, Helm, and elsewhere) showing the problems the current behavior is causing. (List available on request.) is from over 3 years ago. is from 2015.

The old justifications for keeping the current behavior ("works as designed") have proven to be insufficient. It is time for a redesign. The API should accept as meaning "assign a clusterIP" on creation and "keep the current clusterIP" on update. Since, when you set on creation, the resource does not end up with , this means a resource cannot validly have , and since the field is immutable, could never be a correct value for modification under the current API. While this could be considered a breaking change, I doubt there is anyone who would complain about it. Rejecting as a modification is not helping anyone. It is not rejecting an accidental change or avoiding confusion because the meaning is unclear, and it is hard for me to imagine a use case where someone is relying on the current behavior and would be upset about the proposed new behavior (with the possible exception of people writing compliance tests). It is just causing problems for the sake of some kind of theoretical purity that is in the eye of the beholder. Treating as meaning "system should replace this with a valid setting" is equally pure.
cc long-term-issue (note to self)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
keepalive
Is there something I can do to move this forward?
/remove-sig cli /sig network /assign this is sort of the inverse of the issue fixed in Using with a static manifest on an object that has spec fields populated by the API server (like clusterIP in service) or controller (like replicas in deployment) is an anti-pattern. is a better match for this usage. Alternatively, a controller that did a 'read -modify -replace' could be ok, since it would be starting with the current API server state.
This is not impossible to fix in the PUT path (), but it basically turns PUT into PATCH. We have distinct PATCH operations for a reason, and that is almost certainly what you want in this case. should "just work" as a replacement for . Is there a reason why you can't use PATCH or apply?

- while I continue to argue that is wrong here, maybe we should just cave and make it quietly patch the value of clusterIP in this specific case? I really dislike it, not just for purity reasons, but because I have a hard time seeing if that will bite us later. Thoughts?
I really dislike it as well. "replace all spec content" is a bad thing to teach clients to do on resources that have fields set by different actors.
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. - My inclination is to close this. I know it's not what you want, but I do think you and most users will be better served by patch or apply in almost all cases (even knowing that Service has an ambiguity around changing ports that overlap each other which is not easily fixed because it is breaking). Is there a reason why you can't use patch or apply?
This really seems related to bug in Helm, as far as what I have read, and some recent (Jan2021) posts suggest that was fixed. Have not tested this out myself. Agree that adhering to expected PUT/PATCH api behavior would be best. Thanks for the followup!
I reject the idea that this is a bug in Helm, even though Helm is working hard (and failing) to handle the situation. I agree with you when you "Agree that adhering to expected PUT/PATCH api behavior would be best". My point is that the expected "PUT" behavior is that if a "PUT" request succeeds, the exact same "PUT" request given a second time should succeed as well. That is what is broken and I am asking to have fixed. Is there a better reason why I should be prohibited from using replace on a Service, when I can use it for every other resource? Why should Service require special-case handling by the user? I ask you to think of this in the larger context of using Infrastructure as Code (IaC) to manage resources, and the automation around that. In the IaC definition of a Service, the has to be set to in order to get a system-assigned address. Semantically, says "make the resource exactly like this". It can be, and is, used to ensure there is not "drift" in the configuration, i.e.

that the configuration stored in the IaC code base is live. It is the correct thing to use when you want to say "here is the configuration that should be active", and it is what automation uses to reset the configuration after it has drifted. Users should not have to treat Services specially, they should be able to any resources with the same spec that created them. Stating this another way, (A user should be able to the same thing repeatedly, successfully, and without causing changes after the first accepted .) By not accepting for in , you are breaking the contract of making idempotent. (As a matter of practicality, the Helm maintainers are trying to make up for the fact that is not idempotent, but that is not an excuse for Kubernetes to leave this API in its current state.) As I stated above, I am not even advocating for this case to be handled as a . As I see it, the value of is handled specially but not idempotently. The value of currently means one thing if the resource does not exist and something else if it does exist. I am asking instead that it consistently, idempotently be taken as meaning "any valid address: the existing one if one is set, otherwise one randomly assigned by the controller".
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. Unfortunately, to make PUT idempotent in this case, it would need to turn into a variant of PATCH. I'm not even saying I totally disagree, but that is what it means. As for PUT being idempotent, I don't think that's ever really been a hard objective. In this case (and look, I am not saying I am happy about it) that means reading existing state and patching it into the PUT. We can do that (probably) but I can't say for sure that there are no other APIs where this will happen (there probably are not). Note that to do this we would need to fixup , , and probably the newer dual-stack-related fields. Can someone help me understand the "bug"? Is it about using PUT instead of apply? I could see two paths to fixing this, if we want to (and I am still not 100%). 1) Change clusterIP and nodePort fields to pointers.

If not-specified on PUT AND the object exists AND the old warrants a value AND the new warrants a value, copy the existing values. This is a bit tricky for nodePort we'd have to at least specify what happens if the list is reordered or otherwise modified. For example, if the existing object has: and the update specifies: Do I track the move by name? What if the names changed, does that lose the tracking? We've had innumerable issues here because the API-specified merge-key was (is) ambiguous. The only way to get exactly what you want in ALL cases is to RMW the resource. Explicitly setting "" and 0 would work on initialization (compat) but would not be auto-patched. 2) Leave the fields as non-pointers and accept "" and 0 as "DWIM" mode. I dislike this because it makes it less obvious when a user mistakenly tries to clear a field - it used to fail and will now "succeed" but silently change the value. I am not sure I see a clever 3rd option. The only reason I entertain this at all is because of the issues around apply/patch and ports. NOTE: My willingness to think it thru is not a commitment to implement it. I am swamped right now :)
wrote I am shocked that you would think that, although I don't really know what you mean by "hard objective". The doc certainly implies should be idempotent, via such statements as and Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. But that stuff should already be handled by the more generalized resource handling architecture.

The general rule about PUT is that PUT on an existing resource should adjust that resource to be in the same working configuration as if you used that PUT to create a new resource. Calling PUT a second time with the same input should be acceptable and should not change the resource. I believe that Kubernetes generally has a canonical ordering of all fields in a resource (and if there is no clear way to order a list, the canonical order is simply the order in which the list was originally specified on resource creation), so what you do is enumerate the fields of the existing resource in canonical order, filter out the fields like that PUT cannot specify, and then compare it to the PUT resource, after normalizing it into canonical order, and add/delete/replace field by field. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . All we are asking for is that this case be interpreted differently and allowed. is documented as an allowable value in the input, but it is never an allowable value in the realized resource, so it cannot possibly mean to be interpreted as "force this value on the resource". Its meaning is that the system should assign a valid value and the user does not care what it is. So on PUT, when is matched against the exiting , he API should just confirm that is a valid value and, if so, accept the input.
As for PUT being idempotent, I don't think that's ever really been a hard objective. What I meant is it's just never been a primary question in reviewing APIs. Service is a weird case, far outside the norm and I'm not excusing it - just explaining what I meant. Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. For historical reasons, the "key" for ports was not the name but the port value, then the value + protocol. So we can key off that, I think.

I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. Those are all outside of . I don't know of any resource that PUT-PATCH inside the spec. Hilariously, I just noticed that nodePort is even worse - it accepts the non-value and then re-allocates the nodePort. I am going to call this a bug, but now we have a compatibility discussion to have. If a user does not specify a value, we can reasoinably argue that we should retain existing values. But if they specify , do we stay compatible and re-allocate the node port, or do we say that was always a bug? Is it a useful "feature" to be able to re-allocate? We disallow it explicitly gfor ClusterIPs. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. You haven't spent much time in the Service REST code, I guess. I have a large rework of that on-going, but this was not part of it. :) The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . Given the previous point about nodePort, would it be fair to modify your statement as: "when PUT says (or does not specify at all) ..." With the corollart that if you actually specify you will get the current (immutable error) behavior. This seems like it would satisfy helm and other users who are not specifying literal "" - they are not specifying any value at all. Thoughts?
At this point I am willing to call it a bug, albeit an unpleasant one.
wrote I have had regular run-ins with this issue for years, which is why I'm being insistent on getting it fixed properly. I do not recall having similar problems with and so I am less informed on those issues. I don't know what means in the spec or in practice, so I don't have a specific opinion as to what it should do to an existing resource.

I will simply restate the general principle which should apply to just like everything else: what ever you allow as a setting on creation should be allowed in any and should not cause a change if repeatedly applied. The very practical reason for this is so that I can record some place (like a checked into a VCS) that "this is how the resource should be configured" and use that (1) to create the resource and (2) to return the resource to expected configuration after some external operation changed it, using a standard API while otherwise treating the resource as opaque.
/cc
/cc (i know i am going to regret this)
It is a bug. Not because it does not match rest api semantics. But because it is not consistent with the rest of the apis. In other words, If user can do on other resources they should be able to do it on as well. It is hard (and weird) to ask users to do one thing for all resources, but then do a totally different thing for Services. As discussed over sig-network the problem will be user intent. Some ideas for a solution: If irrespective of changes in other we can assume that the user is replacing and does not care about ClusterIP value, hence copying to new should be fine. It is basically like the user saying so copying old=new should be Ok. If that is a failure case, because it breaks ClusterIP immutability rules ( we should really, really remove ClusterIP immutability but i digress). We will still require the validation to work as is. e.g. replacing with with a type that does not require ClusterIP and not setting it to should fail. I can look into building a POC for this.
I am probably missing some subtlety here, but as in I suspect that Helm is behaving properly and the problem is in way Kubernetes handles s: the of a resource should never be rewritten with variable environmental information; that is what is for. In other words could be set in the unusual case that you wished to pick an IP yourself but would normally be null; would reflect the service’s current value, expected to be retained across unrelated patches.
nailed it on the head with his last comment. I do agree that we should not be re-defining the behaviour of a PUT request.

But the inconsistency of how a Service updates the field compared to other resources results in inconsistencies in the PUT behaviour. Once a resource defines fields in the object, normally Kubernetes leaves it unchanged. My understanding is that Kubernetes uses this field to use as a reference point to be the "ideal state of the world", using the field to record the "current state of the resource". The only offending case I've seen in the wild is a , as it updates both the field AND the field. The end result is that a with a defining results in differing behaviour than the rest of the resource APIs. From Helm's point of view, a chart author's only course of action is to a) omit from the manifest and all subsequent updates, or b) if you happened to define in your first install, find the value Kubernetes provided and inject it into your template so that the value Kubernetes provided is preseved. Failing to do so results in a failed PUT () request. I wrote this behaviour down in-depth in this Helm comment: I'd propose an alternative solution than solutions: instead of updating after an IP has been assigned to the Service, use the API, i.e. . It would be a breaking change to the API, but it should bring Services back in line with the rest of the resource APIs, leaving as a user-defined field.
I agree
I'm afraid that breaking an API is not a feasible option
That document does allow for breaking changes under the apiVersion. Instead of making a breaking change to v1 services, you can introduce Service v2. Existing v1 services migrate their spec.clusterIP’s existing value (auto-generated or not) to the new version. spec.clusterIP and status.clusterIP would now match their corresponding relationships, and Kubernetes retains backwards/forwards compatibility.
(If it's not clear whether the API changes document does or doesn't allow the change from last comment, let's raise a separate improvement request against that document).
Does the case is important ? Because here I can read or ...
I have a patch. I'll try to get it out for review tonight.
Has anyone yet checked that the purported fix actually addresses the issue in Helm?
Please do!
I can still see the error but I'm on GKE 1.18, will this only apply to Kubernetes 1.21/22?

It will be part of 1.22 or the next 1.21.X (I don't remember now what is the latest stable)
Can anyone confirm that this issue was actually fixed in k8s version 1.21.4 or 1.22.1 ? I do not see it listed in the release notes/change log
The change was released in the 1.22.0-beta.2, so yes it should be available in 1.22. See I do not believe it was backported to 1.21.
Thanks for confirming and providing the details


Question: cniPlugin.execer = fexec cniPlugin.loNetwork.CNIConfig = mockLoCNI <del> mockLoCNI.On("AddNetworkList", context.TODO(), cniPlugin.loNetwork.NetworkConfig, mock.AnythingOfType("*libcni.RuntimeConf")).Return(&types020.Result{IP4: &types020.IPConfig{IP: net.IPNet{IP: []byte{127, 0, 0, 1}}}}, nil) mockLoCNI.On("DelNetworkList", context.TODO(), cniPlugin.loNetwork.NetworkConfig, mock.AnythingOfType("*libcni.RuntimeConf")).Return(nil) </del> <ins> mockLoCNI.On("AddNetworkList", mock.AnythingOfType("*context.emptyCtx"), cniPlugin.loNetwork.NetworkConfig, mock.AnythingOfType("*libcni.RuntimeConf")).Return(&types020.Result{IP4: &types020.IPConfig{IP: net.IPNet{IP: []byte{127, 0, 0, 1}}}}, nil) mockLoCNI.On("DelNetworkList", mock.AnythingOfType("*context.emptyCtx"), cniPlugin.loNetwork.NetworkConfig, mock.AnythingOfType("*libcni.RuntimeConf")).Return(nil) </ins> // Check that status returns an error if err := cniPlugin.Status(); err == nil {

The upper test heavily flakes in master:
/cc
Hey I think this might have something to do with your changes. Something is definitely a bit off. It probably is a faulty test...
duplicate of
Looks like an occasional failure, and I don't think this is caused by my code change because I run tests for a few times, and sometime it is passed.... Still I can help to locate the root cause.
I revert my code change, test still fails from time to time...
I think i've found out the root cause, I'll pull a request soon, . cc

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: //pkg/kubelet/dockershim/network/cni:godefaulttest Since when has it been failing: Unsure, it doesn't appear in triage, but I see it regularly in PRs. Testgrid link: This package is the top flake currently Reason for failure: Anything else we need to know: /sig node /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
The test results have the following three phenomena. b c


Question: <ins> {% set num_cpus = salt['grains.get']('num_cpus', '') %} </ins> { "apiVersion": "v1beta3", "kind": "Pod",

After recent performance fixes we are now not bound by CPU or disk IOPS. Apiserver during density test uses around 80% of CPU. Still we observe high request latency for pods, events, endpoints and sometimes other resources. I was able to narrow this down to slow etcd. Etcd uses up to 80% of CPU. For some reason we see high latency of requests to etcd. Increasing disk (to increase IOPS) or number of cores doesn't help. In order to meet criteria defined in 1.0 roadmap () we have to make it much faster. Do you have some suggestions how to debug this? Any ideas what could be the reason?
a related question: from the etcd code it seems that etcd is exposing some metrics under /metrics handler: However, when accessing this handler I'm getting 404 error (both with 2.0.9 and 2.0.10 etcd versions). Do you know what could be the reason?
You can try with etcd 2.1.0-alpha0, which gives you a better performance, less CPU usage and the /metrics endpoint (though it is still expr).
Thanks! We will give it a try. Does etcd export pprof data so that we can profile it for lock contention?
Thanks - we will try it. So far I downloaded it locally and indeed it has /metrics endpoint. Thanks!
Not really. If you have time, I think the first thing you can do is to switch the default json encoder to ffjson. Do you know how much performance do you actually need in terms of write/read?
currently we are focusing more on latency. The issue that we observed is that when reading from etcd the whole directory (which is ~3000 entries, each entry of size ~2kB), it can take more than 2 seconds (which seems to be a lot).
Thanks for the suggestion. I will take a look into it.
Yea... That is probably caused by slow json encoding... :(
If that's parsing json shouldn't it mean that there is high CPU usage?
This is what you told me. :) So I highly suspect it is caused by encoding after you gave me more context.
So you're saying that this json encoding happens under lock so that we cannot use more CPU because other requests are blocked. Is my understanding correct?

Can you point us where this encoding happens?
To clarify - by I meant . Sorry for not being clear
The encoding happens without holding any lock: You give etcd multiple cores, and it only uses 80% of one? Is this what you are saying? I still do not quite follow actually.
ffjson seems like a great improvement if the 2x benchmark comparisons vs golang json encoding materialize. Thanks! However I can't help noticing that that even ffjson is more than 30x slower than some protobuf implementations (according to the benchmarks published at ) Have you considered adding a protobuf-based API to etcd to boost performance? Obviously in the short term we might be able to simply "throw cores at the problem", but over time I think that this will become expensive, and a protobuf interface might make sense.
Take a look at:
Beautiful :-)
Let me start from the beginning :) In my test setup master is n1-standard-8 (i.e. 8 cores) machine with 2 PD-SSD disks each 200GB. Under high load generated by density tests total cpu utilization is about 50% (i.e. we use on average 4 cores out of 8). There are two biggest CPU consumers: apiserver -usually up to 3 cores etcd -usually up to 0.8 core (never above 1) With this setup we observe high latency in requests to etcd that list directories with 3000 entries each ~2kB. For some reason etcd does not use all the cpu it could. I tried to look if etcd changes GOMAXPROCS. It seems it doesn't: Maybe this is the reason? As a result etcd cannot consume more than 1 core?
BTW, I would be careful not to conflate JSON and HTTP vs Protobuf and RPC. e.g. One can do Protobuf over HTTP without RPC, and still get most of the performance gains. I'll comment further in the etcd RFC.
Yes. You have to give GOMAXPROCS more cores, then etcd can use more.
We plan to use gRPC, which is protobuf over http2 basically. But we might choose protobuf over plain http with a /rpc endpoint in the end. Not sure yet.
cc
Is it just a regular env variable?

I've only changed this using
Yes. It is.
Perfect! I'll give it a try tomorrow.
This improved stats a lot! The only offending ones (high percentiles above 1s) are listing pods and events: I'll look into this today.
(Just to be clear - this is still on 8 core master with fast drives)
I guess 2 core is enough. Also the read latency has very little thing to do with seek latency. Everything in etcd is in memory for the moment.
Regarding disk - out experiments confirm that seek latency doesn't really matter here. I'm not sure what the difference will be between setting GOMAXPROCS e.g. to 2 and to 4 (we should check it).
so for clarity, did you: export GOMAXPROCS= etcd --myparams ? /cc
Yes. For testing I've just hardcoded this. I'll send PR tomorrow that will automatically set this based on the number of cores. One thing to mention that we still don't use all 8 cores. According to some further debugging now the bottle neck is on apiserver side. I'll debug more tomorrow.


Question: return (len(value) <= LabelValueMaxLength && labelValueRegexp.MatchString(value)) } <del> // Annotation values are opaque. func IsValidAnnotationValue(value string) bool { return true } </del> const QualifiedNameFmt string = "(" + qnameTokenFmt + "/)?" + qnameTokenFmt const QualifiedNameMaxLength int = 253

For example, creating a service with id 'redis-master10012345678999' fails and the returned message contains the following: "message": "service "redis-master10012345678999" is invalid: name: invalid value 'redis-master10012345678999' 'invalid value' is too general, users should understand exactly what is wrong with the input. for example - is it too long? does it contain characters that are not allowed?
Agree - there is now a place to stick the explanations in the ValidationError - we need to link together validation logic and that string. , abonas wrote:
I checked and it seems there are only two errors that do not have detailed explanation. I'll send a PR to fix this.
This is one:
Thanks for the pointer Brian! I'll send second PR to fix this one.
Still second PR to be sent.


Question: } } <ins> // Create and link the kube forward chain. { if _, err := proxier.iptables.EnsureChain(utiliptables.TableFilter, kubeForwardChain); err != nil { glog.Errorf("Failed to ensure that %s chain %s exists: %v", utiliptables.TableFilter, kubeForwardChain, err) return } comment := "kubernetes forward rules" args := []string{"-m", "comment", "--comment", comment, "-j", string(kubeForwardChain)} if _, err := proxier.iptables.EnsureRule(utiliptables.Prepend, utiliptables.TableFilter, utiliptables.ChainForward, args...); err != nil { glog.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", utiliptables.TableFilter, utiliptables.ChainForward, kubeForwardChain, err) return } } </ins> // // Below this point we will not return until we try to write the iptables rules. //

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: 3x Ubuntu 16.04 nodes running in VMware vSphere OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic Install tools: kubeadm Others: CNI: Canal (flannel + calico) What happened: I originally opened to make Canal CNI work when the node's FORWARD chain default policy is set to DROP. However, I realized that the DROP policy was the root cause for why my other CNI tests also were failing. When either weave and flannel CNI is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails. Specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop. Also, the kube-dns pod also crashes. <details<summarykubectl get pods --all-namespaces -o wide</summary</details<details<summarykubectl logs --namespace=kube-system kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kube-dns--gz34l</summary</details<details<summarysudo iptables -t filter -v -L FORWARD --line-numbers (197 currently has the dashboard pod scheduled. Note the 2 dropped packets.

)</summary</details</brWhat you expected to happen: Either it should be better documented that the default FORWARD chain policy should be ACCEPT or rules should be to prevent packets from hitting the default policy. As a note, Ubuntu 16.04 defaults the FORWARD chain to DROP. I assume the is why the packets are going through the FORWARD chain in the first place but without , the dashboard still can not access the kube-apiserver. One option that I suggested in is to have kube-proxy add an iptables FORWARD rule to ACCEPT packets that have been marked by KUBE-MARK-MASQ. How to reproduce it (as minimally and precisely as possible): Create a cluster with: Weave or Flannel CNI enabled on all nodes kube-proxy set to kubernetes-dashboard enabled. kubernetes-dashboard will crash often. Anything else do we need to know: is closed because I thought the issue was specific to Canal and instead opened However, this looks to be a larger issue that kubernetes itself should address.
Weave issue:
I think this is going to hit a lot of people as Docker 1.13 is now setting the FORWARD chain to drop by default if ip_forward is set. See
See also
overlap with ?
I think a solution for this issue or (which is closer to what I feel like a proper solution is) should be considered for inclusion for K8s 1.8 since it will officially support docker 1.13. The problem as I see it is that kube-proxy sets up rules to forward traffic for services (specifically I have seen the problem with NodePorts) yet does not ensure or have any requirement (to my knowledge) that the forwarded traffic be allowed when the traffic is forwarded to a host other than the one receiving the traffic. I observed this when using K8s v1.7.5 installed with kubeadm v1.7.5 and Calico v2.5. It can be remedied with the addition of but hopefully there is a better answer than doing that manually and hopefully something more specific to what is needed. I saw the comment on the Docker 1.13 validation issue linking to the configuration script used in the GCE docker images which is why the e2es are not having any issues.
If we can be more precise about which traffic we require to ALLOW, I'm OK with adding rules to open that.

I've been doing some testing with the first two options and chatted with and about these potential options for adding some forwarding rules. They all are a certainly lot more specific than the previous blanket all ALLOW. any traffic sourced or destined for the clusterCIDR. traffic that has the KUBE-MARK-MASQ set and conntrack rules to accept the follow on traffic. a third option of adding a more specific mark that is only set on specific traffic that it is known will need to be forwarded and match on that (this option would probably need the conntrack rules like the above option too) do any of those options appeal to you or do you see any problems with them?
If we add some more rules, we shouldn't put them in the top-level chain since it makes them hard to manage/clean up. Best to add them to a chain that kube-proxy owns that is jumped to from the top-level chain.
doesn't necessarily know the ClusterCIDR. I could see your proposal making sense in conjunction with .
Very good point I should have mentioned that the options I presented are assuming clusterCIDR is set. If clusterCIDR is not defined then the first option is not possible (except for the wide open option that these rules are attempting to avoid). I guess on options 2 & 3 if it was not set then a conntrack rule could be without the clusterCIDR. I would appreciate feedback on the desired behavior if clusterCIDR is not set too.
The PR that was merged solved the issue for pods inside clusterCIDR. But it doesn't handle hostNetwork pods that use the node IP, like when you want to expose the API server as a load-balanced service externally. I'm not sure why the “kubernetes forwarding conntrack pod source rule” has to be limited to the pod CIDR, since --ctstate RELATED,ESTABLISHED should already ensure it does not allow traffic that wasn't already allowed by another rule.
I'd argue that any iptables/firewalling for a pod using isn't the responsibility of Kubernetes, There's no reasonable way for k8s to actually know what interfaces a host-network container binds to, what ports it listens on, or what expectations it has. The cluster operator should probably be responsible for that networking case.
I meant explicitly regarding services where those pods are the endpoints of a service, so Kubernetes does know the address and the port.

Due to limitation of state RELATED,ESTABLISHED in FORWARD rule i.e. current implementation, the unidirectional UDP traffic apps won't work. Because packets (except first) aren't ESTABILISHED from iptables's perspective (no reply from endpoint, no handshake). Shouldn't this be something like and not
To add to comment, bidirectional UDP apps are brittle as well. Dropping or failing to reply to the first packet will prevent the app from ever receiving a second. More discussion on


Question: Start etcd and verify that it is running: ```bash <del> $ sudo docker run -d --hostname $(hostname -f) --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd </del> <ins> $ sudo docker run -d --hostname $(uname -n) --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd </ins> ``` ```bash

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: ss.LoadBalancerSku = loadBalancerSkuStandard } <del> expectedVMSS := buildTestVMSS([]string{testLBBackendpoolID0}) </del> <ins> expectedVMSS := buildTestVMSS(testVMSSName, []string{testLBBackendpoolID0}) </ins> if test.isVMSSDeallocating { expectedVMSS.ProvisioningState = &virtualMachineScaleSetsDeallocating }

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestEnsureBackendPoolDeletedConcurrently Testgrid link: Reason for failure: /sig cloud-provider
cc
/assign /area provider/azure


Question: "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apiserver/pkg/endpoints/metrics" "k8s.io/apiserver/pkg/server/httplog" <del> "k8s.io/client-go/informers" </del> "k8s.io/klog/v2" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: for extension API server panics with: Apparently introduces this panic with . What you expected to happen: to do not panic. How to reproduce it (as minimally and precisely as possible): extension API server with - for example the endpoint that the apiserver panics with Anything else we need to know?: Environment: Kubernetes version (use ): / Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind regression
/sig api-machinery
/assign
cc
Oops - sorry for that. It seems it is basically this comment: that I blindly applied without e2e retesting (the problem seem to be lack of tests, which is a bit surprising because bugs in my previous implementations were caught by tests). ^^ BTW - this has been cherrypicked to previous releases: 16: 17: 18: so it's actually even more critical...
I can file the automated cherry-pick PRs once the PR against the master branch is merged. :)
Reopening - since we need to cherrypick it still.
I now opened the cherry-picks: 16 - 17 - 18 -
Fix and picks are merged


Question: "//pkg/kubelet/types:go_default_library", "//pkg/kubelet/util/cache:go_default_library", "//pkg/kubelet/util/ioutils:go_default_library", <del> "//pkg/kubelet/util/store:go_default_library", </del> "//pkg/security/apparmor:go_default_library", "//pkg/util/parsers:go_default_library", "//vendor/github.com/armon/circbuf:go_default_library",

/kind bug are failing. seems like the likely cause. cc /assign
ping
seems one test still failing.


Question: package gce_pd import ( <del> "errors" </del> "fmt" "os" "path" <ins> "path/filepath" "strings" </ins> "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider" "github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider/gce" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/exec" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/mount" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/operationmanager" </ins> "github.com/golang/glog" ) <ins> const ( diskByIdPath         = "/dev/disk/by-id/" diskGooglePrefix     = "google-" diskScsiGooglePrefix = "scsi-0Google_PersistentDisk_" diskPartitionSuffix  = "-part" diskSDPath           = "/dev/sd" diskSDPattern        = "/dev/sd*" maxChecks            = 10 maxRetries           = 10 checkSleepDuration   = time.Second ) // Singleton operation manager for managing detach clean up go routines var detachCleanupManager = operationmanager.NewOperationManager() </ins> type GCEDiskUtil struct{} // Attaches a disk specified by a volume.GCEPersistentDisk to the current kubelet. // Mounts the disk to it's global path. <del> func (util *GCEDiskUtil) AttachAndMountDisk(pd *gcePersistentDisk, globalPDPath string) error { </del> <ins> func (diskUtil *GCEDiskUtil) AttachAndMountDisk(pd *gcePersistentDisk, globalPDPath string) error { glog.V(5).Infof("AttachAndMountDisk(pd, %q) where pd is %#vrn", globalPDPath, pd) // Terminate any in progress verify detach go routines, this will block until the goroutine is ready to exit because the channel is unbuffered detachCleanupManager.Send(pd.pdName, true) sdBefore, err := filepath.Glob(diskSDPattern) if err != nil { glog.Errorf("Error filepath.Glob("%s"): %vrn", diskSDPattern, err) } sdBeforeSet := util.NewStringSet(sdBefore...) </ins> gce, err := cloudprovider.GetCloudProvider("gce", nil) if err != nil { return err } if err := gce.(*gce_cloud.GCECloud).AttachDisk(pd.pdName, pd.readOnly); err != nil { return err } <del> devicePaths := []string{ path.Join("/dev/disk/by-id/", "google-"+pd.pdName), path.Join("/dev/disk/by-id/", "scsi-0Google_PersistentDisk_"+pd.pdName), } if pd.partition != "" { for i, path := range devicePaths { devicePaths[i] = path + "-part" + pd.partition } } //TODO(jonesdl) There should probably be better method than busy-waiting here. numTries := 0 devicePath := "" // Wait for the disk device to be created for { for _, path := range devicePaths { _, err := os.Stat(path) if err == nil { devicePath = path break } if err != nil && !os.IsNotExist(err) { return err } } if devicePath != "" { break } numTries++ if numTries == 10 { return errors.New("Could not attach disk: Timeout after 10s") } time.Sleep(time.Second) </del> <ins> devicePath, err := verifyAttached(pd, sdBeforeSet, gce) if err != nil { return err </ins> } // Only mount the PD globally once.

The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.


Question: non-nil error is also returned (possibly along with a partially complete list of resolved endpoints. */ func getResolvedEndpoints(endpoints []string) ([]string, error) { <del> resolvedEndpoints := make([]string, 0, len(endpoints)) </del> <ins> resolvedEndpoints := sets.String{} </ins> for _, endpoint := range endpoints { if net.ParseIP(endpoint) == nil { // It's not a valid IP address, so assume it's a DNS name, and try to resolve it, // replacing its DNS name with its IP addresses in expandedEndpoints ipAddrs, err := net.LookupHost(endpoint) if err != nil { <del> return resolvedEndpoints, err </del> <ins> return resolvedEndpoints.List(), err } for _, ip := range ipAddrs { resolvedEndpoints = resolvedEndpoints.Union(sets.NewString(ip)) </ins> } <del> resolvedEndpoints = append(resolvedEndpoints, ipAddrs...) </del> } else { <del> resolvedEndpoints = append(resolvedEndpoints, endpoint) </del> <ins> resolvedEndpoints = resolvedEndpoints.Union(sets.NewString(endpoint)) </ins> } } <del> return resolvedEndpoints, nil </del> <ins> return resolvedEndpoints.List(), nil </ins> } /* ensureDNSRrsets ensures (idempotently, and with minimum mutations) that all of the DNS resource record sets for dnsName are consistent with endpoints.

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-20T20:37:18Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Local: El Capitan, Remote: GCI Kernel (e.g. ): Local: Darwin ... 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep 1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASEX8664 x86_64 Install tools: gcloud Others: n/a What happened: Provisioned 3 clusters each in different regions (us-central1, asia-east1, europe-west1). Each cluster is in 2 zones (w/ gcloud --zone... --additional-zones...) Provisioned Federation API Server and Controller (1.4.4) based on Verified clusters were ready. Provisioned federated service & deployed backends into each of the cluster. Verified the federated services were created, and all of the regional LB were created, and verified all the LB IPs were working w/ direct requests. DNS provider updated DNS w/ most of the service/zone CNAME records. But A records did not complete. Further investigation for the log points to multiple entries in rrdata w/ the same IP due to the multi-zone deployment: I1029 19:24:24. 1 ] Have recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180.

Need recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180I1029 19:24:24. 1 ] Existing recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180is not equivalent to needed recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180, removing existing and adding needed. After creating the clusters w/ only one zone, the DNS provider worked properly. What you expected to happen: Cloud DNS Provider should update the DNS record properly w/ multi-zone clusters. How to reproduce it (as minimally and precisely as possible): 3 x multi-zone clusters Federation API Server and Controller clusters in to federation federated service and backends Anything else do we need to know: Using GKE and Cloud DNS
/cc
Also hit this. Any workarounds?
Sorry, no direct workarounds available yet. You could convert your multi-zone clusters into multiple single-zone clusters, and federate all of those.
FYI, although I'm sure you're aware - also affects AWS multi AZ clusters (1.6.2)
This seems bad. I'll try to take a look into it.
Fixed by


Question: } } <del> func getTestPod(critical bool, name string, resources v1.ResourceRequirements) *v1.Pod { value := scheduling.SystemCriticalPriority </del> <ins> func getNodeName(f *framework.Framework) string { nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{}) framework.ExpectNoError(err) // Assuming that there is only one node, because this is a node e2e test. framework.ExpectEqual(len(nodeList.Items), 1) return nodeList.Items[0].GetName() } func getTestPod(critical bool, name string, resources v1.ResourceRequirements, node string) *v1.Pod { </ins> pod := &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind:       "Pod",

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: expected:    "westus-2", }, { <ins> description: "scaleSet should get availability zone in lower cases", scaleSet:    "ss", vmList:      []string{"vmssee6c2000000", "vmssee6c2000001"}, nodeName:    "vmssee6c2000000", location:    "WestUS", zone:        "2", faultDomain: 3, expected:    "westus-2", }, { </ins> description: "scaleSet should return error for non-exist nodes", scaleSet:    "ss", faultDomain: 3,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider gets the availability zone from IMDS directly and the zone is usually in lower cases in most regions. But there're chances this is not true in some regions, hence customer's workloads depend on availability zone may not work. What you expected to happen: The availability zone is always in lower cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/assign
Fixed by .


Question: import ( "fmt" <ins> "strings" </ins> "time" . "github.com/onsi/ginkgo"

https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
Looking
Flakes happened when stopping overlapping deployments at clean up step.
You probably need to move the check before the deletionTimestamp check. Just make sure that observedGeneration is updated for the first case.
Because the deployments are overlapping, when deleting one deployment and determining if its RSes are deleted, there might be a race when the other deployment creates an RS
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: import ( "fmt" <ins> "strings" </ins> "time" . "github.com/onsi/ginkgo"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: ginkgo.By("Analyzing results") // FIXME ExpectNoError: this will be implemented in KEP 4438 // liveness probes are called for restartable init containers during pod termination <del> err = results.RunTogether(prefixedName(PreStopPrefix, regular1), prefixedName(LivenessPrefix, restartableInit1)) </del> <ins> err = results.RunTogetherLhsFirst(prefixedName(PreStopPrefix, regular1), prefixedName(LivenessPrefix, restartableInit1)) </ins> gomega.Expect(err).To(gomega.HaveOccurred()) // FIXME ExpectNoError: this will be implemented in KEP 4438 // restartable init containers are restarted during pod termination <del> err = results.RunTogether(prefixedName(PreStopPrefix, regular1), restartableInit1) </del> <ins> err = results.RunTogetherLhsFirst(prefixedName(PreStopPrefix, regular1), restartableInit1) </ins> gomega.Expect(err).To(gomega.HaveOccurred()) }) })

master-blocking: ci-node-e2e Recent failures: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is also similar. The fix for that was bumping the delay time.
Hi Vyom - would this be a good first issue because there is a previous similar solution?
I hit the same failure in my . This seems caused because expects the regular container to start before the postStart: I posted a PR() to add to the postStart. However, there are other approaches such as: Introduce as mentioned . Remove from this test case because this check does not look the primary purpose of this testcase.
The test mentioned by is similar to this flake and doesn't use . I can just remove it. I'm also going to add a comment on RunTogether to indicate it expects ordering to be enforced. That being said, shouldn't PostStart be enforced to run after the container is started? /assign
Hey PRs are more than welcome :)
I've submitted the PR, let me know if I didn't follow any best practices or anything else is required, thanks!
Skipped after was merged. Last CI run passed. /close
Closing this issue. <detailsIn response to : Skipped after was merged. Last CI run passed. </details
/reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry.
Reopened this issue. <detailsIn response to : /reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: podsPerNode: 35, cpuLimits: framework.ContainersCPUSummary{ stats.SystemContainerKubelet: {0.50: 0.12, 0.95: 0.14}, <del> stats.SystemContainerRuntime: {0.50: 0.06, 0.95: 0.08}, </del> <ins> stats.SystemContainerRuntime: {0.50: 0.05, 0.95: 0.07}, </ins> }, // We set the memory limits generously because the distribution // of the addon pods affect the memory usage on each node. memLimits: framework.ResourceUsagePerContainer{ <del> stats.SystemContainerRuntime: &framework.ContainerResourceUsage{MemoryRSSInBytes: 100 * 1024 * 1024}, </del> <ins> stats.SystemContainerKubelet: &framework.ContainerResourceUsage{MemoryRSSInBytes: 70 * 1024 * 1024}, stats.SystemContainerRuntime: &framework.ContainerResourceUsage{MemoryRSSInBytes: 150 * 1024 * 1024}, </ins> }, }, { <del> // TODO(yujuhong): Set the limits after collecting enough data. </del> <ins> cpuLimits: framework.ContainersCPUSummary{ stats.SystemContainerKubelet: {0.50: 0.15, 0.95: 0.20}, stats.SystemContainerRuntime: {0.50: 0.06, 0.95: 0.09}, }, </ins> podsPerNode: 100, <ins> // We set the memory limits generously because the distribution // of the addon pods affect the memory usage on each node. memLimits: framework.ResourceUsagePerContainer{ stats.SystemContainerKubelet: &framework.ContainerResourceUsage{MemoryRSSInBytes: 80 * 1024 * 1024}, stats.SystemContainerRuntime: &framework.ContainerResourceUsage{MemoryRSSInBytes: 300 * 1024 * 1024}, }, </ins> }, } for _, testArg := range rTests {

This is deterministically failing only in kubernetes-e2e-gce-serial with and so on. cc/
Docker 1.11 probably just has a higher resource footprint ()
We moved containerd to the docker cgroup in , and the build started to fail () I'll bump the memory limit now for docker 1.11.
Thanks Yu-ju, this issue got lost in the noise for me.


Question: podLabels := map[string]string{"name": nginxImageName} d := newDeployment(deploymentName, 1, podLabels, nginxImageName, nginxImage, extensions.RollingUpdateDeploymentStrategyType, nil) d.Spec.Paused = true <del> tgps := int64(20) </del> <ins> tgps := int64(1) </ins> d.Spec.Template.Spec.TerminationGracePeriodSeconds = &tgps framework.Logf("Creating paused deployment %s", deploymentName) _, err := c.Extensions().Deployments(ns).Create(d)

https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
dup
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
All of these failures are related to the cleanup phase of the test - pods are terminating but never actually removed. We should probably use a zero grace period for pods.
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: for i := range pod.Spec.Containers { here := &pod.Spec.Containers[i] if here.Name == container.Name { <del> return fmt.Sprintf("spec.containers[%d]", i), nil </del> <ins> if here.Name == "" { return fmt.Sprintf("spec.containers[%d]", i), nil } else { return fmt.Sprintf("spec.containers{%s}", here.Name), nil } </ins> } } return "", fmt.Errorf("container %#v not found in pod %#v", container, pod)

Forked from ... The event generated from kubelet for BoundPod and displayed by kubectl should have Container(s) column ; otherwise, the user might think the entire pod is at the running state, but actually only one or some containers are running. When there are missing event(s) for failed containers, the thing gets worse. For example, in case, only net container is running. $ get events | grep testpd Running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get events Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting created Created with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004 Mon, 29 Dec 2014 20:58:38 +0000 testpd Pod Pending scheduled Successfully assigned testpd to kubernetes-minion- Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting pulled Successfully pulled image kubernetes/pause:latest Mon, 29 Dec 2014 20:58:39 +0000 testpd BoundPod running started Started with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004