Question: <ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = Describe("Etcd failure", func() { framework := Framework{BaseName: "etcd-failure"} BeforeEach(func() { framework.beforeEach() Expect(RunRC(RCConfig{ Client:    framework.Client, Name:      "baz", Namespace: framework.Namespace.Name, Image:     "nginx", Replicas:  1, })).NotTo(HaveOccurred()) }) AfterEach(framework.afterEach) It("should recover from network partition with master", func() { etcdFailTest( framework, "sudo iptables -A INPUT -p tcp --destination-port 4001 -j DROP", "sudo iptables -D INPUT -p tcp --destination-port 4001 -j DROP", false, ) }) It("should recover from SIGKILL", func() { etcdFailTest( framework, "pgrep etcd | xargs -I {} sudo kill -9 {}", "echo 'do nothing. monit should restart etcd.'", true, ) }) }) func etcdFailTest(framework Framework, failCommand, fixCommand string, repeat bool) { // This test requires SSH, so the provider check should be identical to // those tests. if !providerIs("gce", "gke") { By(fmt.Sprintf("Skippingt test, which is not implemented for %s", testContext.Provider)) return } doEtcdFailure(failCommand, fixCommand, repeat) checkExistingRCRecovers(framework) ServeImageOrFail(framework.Client, "basic", "gcr.io/google_containers/serve_hostname:1.1") } // For this duration, etcd will be failed by executing a failCommand on the master. // If repeat is true, the failCommand will be called at a rate of once per second for // the failure duration. If repeat is false, failCommand will only be called once at the // beginning of the failure duration. After this duration, we execute a fixCommand on the // master and go on to assert that etcd and kubernetes components recover. const etcdFailureDuration = 20 * time.Second func doEtcdFailure(failCommand, fixCommand string, repeat bool) { By("failing etcd") if repeat { stop := make(chan struct{}, 1) go util.Until(func() { defer GinkgoRecover() masterExec(failCommand) }, 1*time.Second, stop) time.Sleep(etcdFailureDuration) stop <- struct{}{} } else { masterExec(failCommand) time.Sleep(etcdFailureDuration) } masterExec(fixCommand) } func masterExec(cmd string) { stdout, stderr, code, err := SSH(cmd, getMasterHost()+":22", testContext.Provider) Expect(err).NotTo(HaveOccurred()) if code != 0 { Failf("master exec command, '%v' failed with exitcode %v: ntstdout: %vntstderr: %v", cmd, code, stdout, stderr) } } func checkExistingRCRecovers(f Framework) { By("assert that the pre-existing replication controller recovers") podClient := f.Client.Pods(f.Namespace.Name) rcSelector := labels.Set{"name": "baz"}.AsSelector() By("deleting pods from existing replication controller") pods, err := podClient.List(rcSelector, fields.Everything()) Expect(err).NotTo(HaveOccurred()) Expect(len(pods.Items) > 0).Should(BeTrue()) for _, pod := range pods.Items { err = podClient.Delete(pod.Name, api.NewDeleteOptions(0)) Expect(err).NotTo(HaveOccurred()) } By("waiting for replication controller to recover") expectNoError(wait.Poll(time.Millisecond*500, time.Second*30, func() (bool, error) { pods, err := podClient.List(rcSelector, fields.Everything()) Expect(err).NotTo(HaveOccurred()) for _, pod := range pods.Items { if api.IsPodReady(&pod) { return true, nil } } return false, nil })) } </ins>

We ran into a lot of issues when etcd crashes. We need tests to ensure all components / daemons are doing the right things etcd is down, and recover later.
Note that, like the existing test, this is a case of a system level that probably can't be parallelized very easily. (Though rolling over etcd while running other tests is a perfectly fine chaos-monkey style test, but it's not very controlled.) cc
As context, this came up at the conference last week. In particular, if etcd crashes or goes unavailable, then recovers, it was reported that some cluster components do not recover gracefully.
We're still looking for a volunteer for this. Any takers? cc: who might be interested.
has accepted
I discussed the test scenario with We came up with the following: up a few pods, so that the system has something to do. a bit, so that they are initialized up 100 watches on the API server (to simulate a large system) if possible, use NewEndpointController (from - which is supposed to be used by all controllers) to simulate real clients. This EndpointController will automatically restart watches. It's interesting to see what happens at the API server when all watches are restarted at the very same time. etcd (ssh onto the machine) bit (?) new pods (maybe don't wait, just retry on error) to see that they are scheduled (we check here only that the scheduler still works). It makes no sense to test everything after etcd is restarted because we would end up repeating all tests. We will just use the fact that tests are run in random order so we hope next tests will find components that didn't recover properly after etcd crashes (or rather after their watches having been terminated). the test
May I suggest a perhaps simpler first step, which does most of the above with minimal effort... Clone an existing e2e test (guestbook and rc are good candidates) Tweak it slightly such that: -- At the point where success would usually be declared, instead of declaring success, ssh into the etcd machine and render etcd temporarily uncontactable (more on how to do that below). -- Now kill all of the Pods in the application (with appropriate retries). We now rely on the replication controller to recover and reconstitute the test application.

-- check that the test eventually fulfills success criteria again. I think that the above will be super-simple to get working. Once that is working, you can do the same thing for a much larger scale test (e.g. one of the scalability e2e tests.
Oh, and regarding how to make etcd uncontactable for a while, two suggestions: into the node (use the new ssh utility in test/e2e/utils) and schedule a simple cron job to insert an iptables DROP rule to block all incoming traffic to etcd, sleep for a while (I think it will have to be 5 minutes to trigger stuff), and then remove the iptables rule to allow traffic to flow again. alternatively: into the node and schedule a cron job to repeatedly kill the etcd process for ~5 minutes, and then stop killing it, allowing monit to restart it automatically.
I think your step 1/2 is not really necessary-- there are pods running in the cluster by default. black holeing traffic to etcd seems like a good thing to try too, but I think we should also test killing/restarting it.
agreed. Change "alternatively" to "in addition" above :-)
Ok, how about the guestbook test, with just killing etcd before creating all the objects (maybe several times during 5 minutes). Then I'll add more complicated scenarios. I suggest being careful about modifying the environment (like iptables or cron). If the test crashes in the middle, all following tests will fail. Do we have a cleanup/healthcheck executed procedure before each test? I could insert resetting iptables (or cron) there. (If not, shall we create it?)
Yes, just killing etcd before the test will also work. Regarding modifying, yes, that's why I suggested cron (on the etcd machine), so that the cleanup will happen even if the test code itself fails. I think that's good enough.
/subscribe
fwiw we regularly do this, but have chosen not to make it an e2e due to sec-keys and our etcd cluster resides on a different loc from our master. We could use etcd-discovery service to make it more dynamic, but we would still have the issue of keys.
Is anyone currently working on this?

I emailed earlier today to see if he'd be interested in working on it, but since he'll be out Thu-Fri it would probably be better for you to do it, if you're interested.
please take it as I'm OOO next two days.
Just submitted PR to resolve this. I would appreciate review.
Current e2e tests has two failure cases: network partition and SIGKILL. The network partition runs consistently but the SIGKILL is flaky. I'm investigating further.


Question: p.PostBind = appendPluginSet(p.PostBind, src.PostBind) p.Unreserve = appendPluginSet(p.Unreserve, src.Unreserve) } <ins> // Apply merges the plugin configuration from custom plugins, handling disabled sets. func (p *Plugins) Apply(customPlugins *Plugins) { if customPlugins == nil { return } p.QueueSort = mergePluginSets(p.QueueSort, customPlugins.QueueSort) p.PreFilter = mergePluginSets(p.PreFilter, customPlugins.PreFilter) p.Filter = mergePluginSets(p.Filter, customPlugins.Filter) p.PostFilter = mergePluginSets(p.PostFilter, customPlugins.PostFilter) p.Score = mergePluginSets(p.Score, customPlugins.Score) p.Reserve = mergePluginSets(p.Reserve, customPlugins.Reserve) p.Permit = mergePluginSets(p.Permit, customPlugins.Permit) p.PreBind = mergePluginSets(p.PreBind, customPlugins.PreBind) p.Bind = mergePluginSets(p.Bind, customPlugins.Bind) p.PostBind = mergePluginSets(p.PostBind, customPlugins.PostBind) p.Unreserve = mergePluginSets(p.Unreserve, customPlugins.Unreserve) } func mergePluginSets(defaultPluginSet, customPluginSet *PluginSet) *PluginSet { if customPluginSet == nil { customPluginSet = &PluginSet{} } if defaultPluginSet == nil { defaultPluginSet = &PluginSet{} } disabledPlugins := sets.NewString() for _, disabledPlugin := range customPluginSet.Disabled { disabledPlugins.Insert(disabledPlugin.Name) } enabledPlugins := []Plugin{} if !disabledPlugins.Has("*") { for _, defaultEnabledPlugin := range defaultPluginSet.Enabled { if disabledPlugins.Has(defaultEnabledPlugin.Name) { continue } enabledPlugins = append(enabledPlugins, defaultEnabledPlugin) } } enabledPlugins = append(enabledPlugins, customPluginSet.Enabled...) return &PluginSet{Enabled: enabledPlugins} } </ins>

What would you like to be : Actually disable scheduler plugins according to the disabled list in the scheduler's component config. The semantics are defined here: Part of /sig scheduling /priority important-soon /assign
The following semantics seems vague, and I think we should reach consensus on some cases when disabled plugins are involved. Suppose we have two default plugins, and , the runs before , here are the cases we could meet with the scheduling framework, plugins which run after all the default plugins plugins which should run ahead of the plugins plugin which should run ahead of all the plugins -enable default or custom plugin should return errors
Do you have any other cases not mentioned here?
Those cases sound right, I don't have others.
Is this actually what we want? I don't think so. The comment needs rewording.
I plan to revisit the comments after


Question: } // Setup cadvisor to check the container environment <del> c, err := cadvisor.New(0 /*don't start the http server*/, "docker", "/var/lib/kubelet") </del> <ins> c, err := cadvisor.New("", 0 /*don't start the http server*/, "docker", "/var/lib/kubelet") </ins> if err != nil { return printError("Container Runtime Check: %s Could not start cadvisor %v", failed, err) }

cAdvisor provides potentially sensitive data and there's currently no way to block access with anything other than iptables.
What type of cluster are you running (GKE, or what instructions did you use to setup your cluster?) A short term solution might be to try adding flag to kubelet and restarting the kubelet service and see if the sensitive information is no longer leaked. Doing this may the UI and heapster which expect to be able to poll the cadvisor port. I think probably knows if there is a longer term solution planned, such as: only serve on specified interface. if this is localhost, then it breaks the ui, but maybe that is okay. maybe has thought about this. require authentication/authorization to connect to cAdvisor port.
Self deployed cluster, the kubelet is started via
We are working on improving the kubelet APIs which will subsume cadvisor APIs. Until then setting cadvisor port to '0' is probably the way to go.
Is there an issue and target date for doing this?
Filed . The current plan is to complete this for v1.1. cc
Do you mean the read-only kubelet port ( by default IIRC)? If so, then we really need to disable this & provide authorisation options on the kubelet to be able to retrieve stats fro the kubelet via the normal port with a restricted token/cert/auth options. We've discussed this for OpenShift (openshift/origin) too.
Sorry I didn't realise that cadvisor was bound to port 4194 as well. Thought this was all embedded inside the kubelet process without exposing any ports.
Any reason why we start up cadvisor http server at all? Seems unnecessary now kubernetes embeds cadvisor & exposes via & ?
does not expose all the containers (, , etc). is working on introducing a better stats API in kubelet () that should help us shut down cadvisor's http server in kubelet.
Couldn't we just add cadvisor as a handler in the kubelet anyway? No need for separate http server & would mean access to raw data is available if anyone wants it.
Indeed that is an option. But we need a better stats API in kubelet anyways for reasons mentioned in .
Absolutely agree but thinking quick win.
+1 on a quick win.

We just disabled it by default instead: Which reminds me it would be great to have so we could make it easy to turn on/off stuff like this w/o rewriting the entire command-line in a service file.
cc
it's not 100% clear to me whether or not made it safe to use , could you please clarify? On a fresh k8s 1.6.4 cluster still exposes cAdvisor to the world: the file you referenced was deleted as part of coreos/coreos-overlay the documentation @ (CoreOS docs) doesn't mention cAdvisor should be disabled either. Another reason for my question to Vish.
Ahh. That's a bug that the new component config based scheme introduced. being is a valid setting.
I think we can change this. The KubeletConfiguration type is still technically alpha, so it should be ok to make this field a pointer and keep the current default when it's not provided (but allow 0). This won't affect anyone who presently omits the flag and expects port 4194. Theoretically people could be explicitly passing 0 and expecting 4194... I guess... but that's sort of an insane case, and I'd consider that behavior a bug.
Thanks for clarifying guys :+1: The original problem remains though: enabling the cAdvisor port makes the kubelet listen on all interfaces.
Yeah. We can get cAdvisor to bind to the interface that kubelet uses. cc , Antoine Cotten <:
/assign


Question: var _ volume.VolumePlugin = &secretPlugin{} <del> var wrappedVolumeSpec = volume.Spec{ Volume: &api.Volume{VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{Medium: api.StorageMediumMemory}}}, </del> <ins> func wrappedVolumeSpec() volume.Spec { return volume.Spec{ Volume: &api.Volume{VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{Medium: api.StorageMediumMemory}}}, } </ins> } func getPath(uid types.UID, volName string, host volume.VolumeHost) string {

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Prow link to the 2 jobs: * Additional Context:
/sig storage cc
I'm working on , it's hard to get a job yaml right at the first time. I'll check iscsi. /assign
/triage accepted /kind failing-test
ci-kubernetes-e2e-storage-kind-disruptive fixed: has 1 green run pull-kubernetes-e2e-storage-kind-disruptive fixed: , it fails when it should fail, it's green otherwise, I'll continue with iscsi.
/reopen it's still red, but at least it installs now:
Reopened this issue. <detailsIn response to : /reopen it's still red, but at least it installs now: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
e2e-gce-iscsi-serial job passes now: e2e-gce-iscsi is getting greener: I'll let it run for few more days to see if today's failures are flakes. /reopen
Reopened this issue. <detailsIn response to : It's getting greener: /reopen </details
iscsi-serial is green:


Question: return nil } <del> list, err := c.lister.PodPresets(pod.GetNamespace()).List(labels.Everything()) </del> <ins> list, err := c.lister.PodPresets(a.GetNamespace()).List(labels.Everything()) </ins> // Ignore if exclusion annotation is present if podAnnotations := pod.GetAnnotations(); podAnnotations != nil {

/kind bug What happened: A pod with an empty namespace field was mutated by a pod preset in a different namespace. What you expected to happen: Pod presets should only operate on pods in the same namespace. How to reproduce it (as minimally and precisely as possible): Submit a pod with an empty namespace field and observe pod preset mutation. Anything else we need to know?: This is a fix for an alpha non-default feature. Fix submitted in
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery


Question: "//pkg/kubelet/apis/cri:go_default_library", "//pkg/kubelet/apis/cri/v1alpha1/runtime:go_default_library", "//pkg/kubelet/apis/kubeletconfig:go_default_library", <del> "//pkg/kubelet/apis/kubeletconfig/v1alpha1:go_default_library", </del> "//pkg/kubelet/cadvisor:go_default_library", "//pkg/kubelet/certificate:go_default_library", "//pkg/kubelet/cm:go_default_library",

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: getReactor:                 nil, onRepeatedHeartbeatFailure: nil, expectErr:                  false, <ins> client:                     fake.NewSimpleClientset(node), </ins> }, { desc: "connection errors",

Some external controller deleted node object while kubelet was alive Due to ignoring errors in kubelet's lease logic, it created lease without ownerReference set Kubelet doesn't create lease without ownerReference set in any case, even pathological one Some external controller deletes node object Garbage collector properly deletes associated lease object Kubelet tries to update object and it fails on Conflict (saying that UID doesn't match): We go to branch, call backoffEnsureLease and assign result to base, but backoffEnsureLease may return potentially invalid object if ensureLease returned error (which happens when newLeasePostProcessFunc returns error, e.g. when node doesn't exist). We The next loop calls c.newLease on the incorrect base and ignores error in line 181 (so the second attempt to call newLeasePostProcessFunc which also fails with an error) We do UPDATE with incomplete (without effect of newLeasePostProcessFunc) object Given that leases allow CreateOnUpdate a new object, without ownerReference is created: No response <details21.10 </details<details</details<details</details<details</details<details</details<details</details
/sig node
/assign
/cc
/triage accepted need to understand what is the right behavior here
We are in trouble for this problem too. I think It’s exactly as said. Lease has the special update endpoint which can create the new lease objects not only update them. If it receives a request with a new lease object which has non-nil UID field, returns a conflict error. This situation can be occured in the function when a lease object is deleted. The lease object is deleted when the node resource is deleted in the result of GC. When received a conflict error in updating a lease object and the parent node object doesn't exist, , and continue next loop. In the next loop, the function returns the new lease object without ownerReference and UID. As a result, it is created unexpectively by the special update endpoint.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: } mockVMSSVMClient.EXPECT().List(gomock.Any(), ss.ResourceGroup, testVMSSName, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() <ins> mockVMsClient := ss.cloud.VirtualMachinesClient.(*mockvmclient.MockInterface) mockVMsClient.EXPECT().List(gomock.Any(), gomock.Any()).Return([]compute.VirtualMachine{}, nil).AnyTimes() </ins> powerState, err := ss.GetPowerStatusByNodeName("vmss-vm-000001") assert.Equal(t, test.expectedErr, err, test.description+", but an error occurs") assert.Equal(t, test.expectedPowerState, powerState, test.description)

What happened: When the standalone (or VMAS) instance backing a registered node is removed (from Azure, ie. the instance is deleted), that node will remain forever due to the Azure cloud provider only checking whether the node was removed from a containing VMSS. Moreover this will triggers frequent VMSS full re-scans (when the instance name conforms to the expected VMSS instances name format, as we cache miss on every call), hammering Azure's VMSS List API, and causing throttling. A VMAS/standalone node leaving a cluster would cause controller-manager to log that error (here on k8s v1.18.2): What you expected to happen: The cloud provider should answer CM node lifecycle controller that the VM instance went missing and the node can be garbage collected. How to reproduce it (as minimally and precisely as possible): 1- Create a non-VMSS Azure node 2- Configure it to register to an Azure k8s cluster 3- Delete that node's instance from Azure console Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.2 Cloud provider or hardware configuration: Azure, controller-manager running with and OS (e.g: ): Ubuntu 20.04 LTS Kernel (e.g. ): Linux 5.4.0-1010-azure Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure


Question: return true } <ins> // User can upgrade (add another clusterIP or ipFamily) or can downgrade (remove secondary clusterIP or ipFamily), // but CAN NOT change primary/secondary clusterIP || ipFamily UNLESS they are changing from/to/ON ExternalName // so not care about order, only need check the length. if len(oldService.Spec.IPFamilies) != len(newService.Spec.IPFamilies) { c.eventRecorder.Eventf(newService, v1.EventTypeNormal, "IPFamilies", "Count: %v -> %v", len(oldService.Spec.IPFamilies), len(newService.Spec.IPFamilies)) return true } </ins> return false }

Currently, when loadbalancer service's ipFammilies updatedservice controller will ignore this event. That means cloud provider can't update ipv4/ipv6 backend. So, I'm thinking service controller need add watching 'service's ipFammilies updated' event, It'd be important for cloud provider update dual-stack backend. code at Our load balancer supports dual-stack. When the services in the cluster are updated from IPv4 to IPv6(same ipv6-ipv4ipv4-ipv4/ipv6), it is expected that the cloud provider can perceive this event and update the IPv4/IPv6 address to the backend of the load balancer. /sig cloud-provider /area cloudprovider
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah, that is right, the logic need to be adapted /kind bug /sig network /sig cloud-provider


Question: estimate := int64(0) glog.V(4).Infof("namespace controller - deleteAllContent - namespace: %s, gvrs: %v", namespace, groupVersionResources) // iterate over each group version, and attempt to delete all of its resources <ins> // we sort resources to delete in a priority order that deletes pods LAST sort.Sort(sortableGroupVersionResources(groupVersionResources)) </ins> for _, gvr := range groupVersionResources { gvrEstimate, err := deleteAllContentForGroupVersionResource(kubeClient, clientPool, opCache, gvr, namespace, namespaceDeletedAt) if err != nil {

I've noticed at least 10 separate test flake issues open in which at least one failure looks like this: Current list: &q=is%3Aissue%20label%3Akind%2Fflake%20is%3Aopen%20%22was%20not%20deleted%20within%20limit%22 The 10 current issues (note that some are 'broken test run' issues and might have multiple failure causes): cc
Just to clarify, the pods were deleted in the test (hence the message). I mentioned this before in
I was chasing down another p0 but now that that is resolved will try to look for more that we can do here. On Wednesday, July 27, 2016, Yu-Ju Hong wrote:
- why did you close this - this is happening quite frquently.
Thanks for reopening! We will now automatically choose and owner for auto-opened flake issues. I closed a bunch of old flakes that were created with no assignee. I want them to get recreated with an assignee. The hope is someone will actually look at these.
is another example of this flake.
has the same symptom too.
One more example: would it help if we bump the timeout from 5 min to N (e.g., 8) min? or should we ask tests that create significant number of pods to clean up after themselves?
I am going to try and look at optimizing the requeue behavior for tests that have a lot of pods and we are waiting for them to delete. Looks like once they delete, we don't observe their deletion until the next resync
Opened a PR to improve e2e framework ns deletion to avoid duplicate ns delete operations, and to give more detailed information on what content did not actually get deleted.


Question: } } <ins> func TestCRDsOnStartup(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ValidatingAdmissionPolicy, true)() testContext, testCancel := context.WithCancel(context.Background()) defer testCancel() // Start server and create CRD, and validatingadmission policy and binding etcdConfig := framework.SharedEtcd() server := apiservertesting.StartTestServerOrDie(t, nil, []string{ "--enable-admission-plugins", "ValidatingAdmissionPolicy", "--authorization-mode=RBAC", "--anonymous-auth", }, etcdConfig) client := clientset.NewForConfigOrDie(server.ClientConfig) dynamicClient := dynamic.NewForConfigOrDie(server.ClientConfig) apiextclient := apiextensionsclientset.NewForConfigOrDie(server.ClientConfig) myCRD := &apiextensionsv1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name: "foos.cr.bar.com", }, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ Group: "cr.bar.com", Scope: apiextensionsv1.NamespaceScoped, Names: apiextensionsv1.CustomResourceDefinitionNames{ Plural: "foos", Kind:   "Foo", }, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema:  fixtures.AllowAllSchema(), }, }, }, } // Create a bunch of fake CRDs to make the initial startup sync take a long time for i := 0; i < 100; i++ { crd := myCRD.DeepCopy() crd.Name = fmt.Sprintf("foos%d.cr.bar.com", i) crd.Spec.Names.Plural = fmt.Sprintf("foos%d", i) crd.Spec.Names.Kind = fmt.Sprintf("Foo%d", i) if _, err := apiextclient.ApiextensionsV1().CustomResourceDefinitions().Create(context.Background(), crd, metav1.CreateOptions{}); err != nil { t.Fatal(err) } } etcd.CreateTestCRDs(t, apiextclient, false, myCRD) crdGVK := schema.GroupVersionKind{ Group:   "cr.bar.com", Version: "v1", Kind:    "Foo", } crdGVR := crdGVK.GroupVersion().WithResource("foos") param := &unstructured.Unstructured{ Object: map[string]interface{}{ "metadata": map[string]interface{}{ "name":      "test", "namespace": "default", }, "foo": "bar", }, } param.GetObjectKind().SetGroupVersionKind(crdGVK) if _, err := dynamicClient.Resource(crdGVR).Namespace("default").Create(context.TODO(), param, metav1.CreateOptions{}); err != nil { t.Fatal(err) } policy := withValidations([]admissionregistrationv1beta1.Validation{ { Expression: "object.metadata.name.startsWith(params.metadata.name)", Message:    "wrong prefix", }, }, withParams(withCRDParamKind(crdGVK.Kind, crdGVK.Group, crdGVK.Version), withNamespaceMatch(withFailurePolicy(admissionregistrationv1beta1.Fail, makePolicy("allowed-prefixes"))))) policy = withWaitReadyConstraintAndExpression(policy) _, err := client.AdmissionregistrationV1beta1().ValidatingAdmissionPolicies().Create(context.TODO(), policy, metav1.CreateOptions{}) if err != nil { t.Fatal(err) } // validate that namespaces starting with "test" are allowed policyBinding := makeBinding("allowed-prefixes-binding", "allowed-prefixes", "test") if err := createAndWaitReady(t, client, policyBinding, nil); err != nil { t.Fatal(err) } doCheck := func(client clientset.Interface) { if waitErr := wait.PollUntilContextTimeout(testContext, time.Millisecond*100, 3*time.Minute, true, func(ctx context.Context) (bool, error) { disallowedNamespace := &v1.Namespace{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "not-test-", }, } _, err = client.CoreV1().Namespaces().Create(testContext, disallowedNamespace, metav1.CreateOptions{}) if err == nil { return false, nil } if strings.Contains(err.Error(), "not yet synced to use for admission") { return false, nil } if strings.Contains(err.Error(), "failed to find resource referenced by paramKind") { return false, nil } if !strings.Contains(err.Error(), "wrong prefix") { return false, err } return true, nil }); waitErr != nil { t.Errorf("timed out waiting: %v", err) } } // Show that the policy & binding are correctly working before restarting // to use the paramKind and deliver an error doCheck(client) server.TearDownFn() // Start the server. server = apiservertesting.StartTestServerOrDie( t, &apiservertesting.TestServerInstanceOptions{ SkipHealthzCheck: true, }, []string{ "--enable-admission-plugins", "ValidatingAdmissionPolicy", "--authorization-mode=RBAC", "--anonymous-auth", }, etcdConfig) defer server.TearDownFn() // Now that the server is restarted, show again that the policy & binding are correctly working client = clientset.NewForConfigOrDie(server.ClientConfig) doCheck(client) } </ins> type clientFn func(t *testing.T, adminClient *clientset.Clientset, clientConfig *rest.Config, rules []rbacv1.PolicyRule) *clientset.Clientset func secondaryAuthorizationUserClient(t *testing.T, adminClient *clientset.Clientset, clientConfig *rest.Config, rules []rbacv1.PolicyRule) *clientset.Clientset {

I'm using ValidatingAdmissionPolicy(VAP) with setting my own CRD to paramKind in my on-premise cluster. It works fine, but once I restarted one of the kube-apiserver instances it changed to reject both valid and invalid requests with the error message like: VAP using CRDs as paramKind should continue to work fine even when one of kube-apiserver instances is restarted. You can reproduce it locally using kind cluster by following procedure. <details</detailsI recovered this failure situation by mutating field of all existing VAPs and running reconciliation of this field explicitly. I've also tried to mutate , but it is not enough. Without the above operation, all requests are rejected. I might be able to avoid this error by setting to Allow, but I don't want to do that to enforce this validation check. v1.28.3, v1.29.0 <details</detailsN/A N/A N/A N/A N/A
/sig api-machinery
/triage accepted
We are experiencing the same issue with Kubernetes v1.27.9, but it is very flaky and does not occur all the time. Edit 2: it's totally flaky and cannot be mitigated with the feature gate . Sometimes a single restart of the kube-apiserver fixes this bug again, sometimes it needs multiple restarts.
cc I don't think we should go to GA without a fix for this.
How long does this condition last for you? The param CRD is checked when the policy is resynced as you said. The controller is written so that if an error is returned from looking up the param resource it retries again later, so the manual resync shouldn't be necessary. But it does use exponential backoff: I'd be curious if you have more logs, specifically around lines from the controller automatic resync that include "error syncing" Additionally, the RESTMapper used to perform this check is refreshed every 30 seconds, so there could be up to a 30s delay at least after a CRD is that it becomes visible (assuming no rate limit):
/assign Thanks for taking this! Thanks for raising the issue. Would you mind sharing more info on this one especially how long/often this happens, any log to share, is the issue still there after waiting for longer, etc after Alex shared more details above? Thank you!

Thank you for your comments! I have reproduced this problem again with the procedure as I described. As you know, this problem is flaky and it recurred 3 times in 5 trials. When it recurs it continues for over an hour. I’m not sure how long it will continue, I'm currently running the test for a long time to investigate it. I will let you know when I get the results. (Probably, it will continue until next resync will be performed.) Here is the log around the when the problem is recurring. If needed, I will get more detailed log and share it with you.
I kept running the test for over a day, and it continued to happen throughout the test. I tried to recover it by patching the 'spec.paramKind' field like following, but it didn’t work in the 1.29 kind environment. (This worked in our 1.28.3 on-premise cluster, so there might be some difference between these versions.). After all, I deleted and recreated the validatingAdmissionPolicy then it recovered to execute validation correctly.
Thanks for your detailed report! From this I think I may have found the source of the issue. It was curious to me that your logs only contained a single occurrance of: This indicates to me that it was not retrying correctly. Snooping around the code I see this line to the top of : This short circuits the resync since the spec did not change. I was able to reproduce your bug report in an integration test, and with a small code change I now see the policy syncing retried multiple times (with exponential backoff) until success after restarting the apiserver: code change and test here:
thanks a lot for the fix. Unfortunately, it will take a lot of time until this arrives to the distro that we are using. As we already rely on VAP for some policy enforcement, we would like to understand if there would be any simple workaround that can help when this issue happens (apart from restart the API several times when it gets fixed...).
backport PR is already created, see
thanks. Yes, just realized it after I wrote my comment. Until that arrives to the distros, what would be the simplet/best way to "trigger" a sync manually to "get rid of the issue" when it happens?

Seems deleting the VAP, VAPB and CRD and re-apply them removes the issue... Is there any better way?


Question: Namespace: "", } <del> var healthzServer *healthcheck.ProxierHealthServer </del> <ins> var healthzServer healthcheck.ProxierHealthUpdater </ins> if len(config.HealthzBindAddress) > 0 { healthzServer = healthcheck.NewProxierHealthServer(config.HealthzBindAddress, 2*config.IPTables.SyncPeriod.Duration, recorder, nodeRef) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.

/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved


Question: ) func TestSchedulerDefaults(t *testing.T) { <ins> enable := true </ins> tests := []struct { name     string config   *kubeschedulerconfigv1alpha1.KubeSchedulerConfiguration

What would you like to be : Enable profiling by default in the scheduler. We can do this by defaulting the CC flags to true. Why is this needed: It is useful when debugging the performance of a running cluster. API server already enables profiling by default. /sig scheduling /help /milestone v1.17
/assign /remove-help


Question: <del> /* Copyright 2014 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // Package label created persistent volumes with zone information // as provided by the cloud provider package label // import "k8s.io/kubernetes/plugin/pkg/admission/storage/persistentvolume/label" </del>

In 1.30, the PersistentVolumeLabel admission plugin blocks creation of in-tree AzureDisk and vSphere PVs: The reason is that the cloud provider was removed (), but the admission plugin still calls GetCloudProvider there, which fails: The PV should be admitted + created. Enable PersistentVolumeLabel admission plugin. For example, when running local-up- Create AzureDisk PV on any 1.30 cluster, you don't need Azure cloud: /sig storage /priority important-soon <detailsToday's master </details<detailsAzure </detailsNo response <detailslocal-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am fixing the worst bug in and I plan to backport it to 1.30. However, we should consider removing the whole PersistentVolumeLabel admission plugin. Most of the cloud providers were removed, only GCE remains and IMO even that one will be removed sooner or later. The admission plugin has no way to get volume zone + region and is therefore useless. The admission said (in 1.26, where we had all in-tree cloud providers): Such webhook never materialized for AWS, Azure nor vSphere and I am not aware of any plans for GCE.
It did materialize in So removing it from k/k is the best way forward.


Question: } // createdTokenSecret returns the ServiceAccountToken secret posted when creating a new token secret. <del> // Named "default-token-p7w9c", since that is the first generated name after rand.Seed(1) </del> <ins> // Named "default-token-stdpg", since that is the first generated name after rand.Seed(1) </ins> func createdTokenSecret(overrideName ...string) *v1.Secret { <del> return namedCreatedTokenSecret("default-token-p7w9c") </del> <ins> return namedCreatedTokenSecret("default-token-stdpg") </ins> } // namedTokenSecret returns the ServiceAccountToken secret posted when creating a new token secret with the given name.

/kind feature What happened: In playing with the nginx ingress controller deployment, I noticed the following pod name: nginx-ingress-controller--c0ck5 I figure we should possibly put something in place to prevent accidental l33t profanity ;-). What you expected to happen: The following pod name: nginx-ingress-controller--r0ck5
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
My first github (k8s) issue. I've no idea which SIG would cover this topic. /sig product-management
/sig api-machinery
Previously we removed vowels in . We can 3xt3nd the idea with 0, 1 (can stand for ) and 3. would you like to send a PR? If not, I can work on this.
/assign jamesawebb1
Reassign to me if you do not want to contribute a PR.


Question: "//vendor/github.com/spf13/pflag:go_default_library", "//vendor/k8s.io/klog:go_default_library", "//vendor/k8s.io/utils/exec:go_default_library", <del> "//vendor/k8s.io/utils/nsenter:go_default_library", </del> ] + select({ "@io_bazel_rules_go//go/platform:linux": [ "//vendor/github.com/sigma/go-inotify:go_default_library",

There was a previous attempt : We have open issues that is not being worked on: I think it's time to delete the CI jobs and start the deprecation cycle. /sig node /assign
We discussed this at today's SIG Node meeting. SIG Node agreed to deprecate "containerized kubelet" with the following action items: The project was initiated for self-host cluster by SIG cluster lifecycle. We should communicate with them first to understand the potential dependencies. There were some demand from the community for this feature in the past, but we don't know who are using this in their production. We should announce this deprecation at Kubernetes Community weekly meeting and send email to kubernetes-dev
will do thanks
/sig storage /sig cluster-lifecycle
cc cc
Does this deprecate based in any way? I have a project that depends on this and I think it is a real use case for a container based . Running either a based or a dedicated container is key to our design.
it doesn't. the main thing to go would be this flag, if you are not using it, you are good. [1]
What is the reason behind this decision? We are heavily using containerized in production running on CoreOS. Would that mean that we no longer can run kubelet in a container?
Do you use the flag as shown here? Do you have mounts as shown just below the line pointed above If so, you have about a 3 releases per our deprecation policies to switch over to some other way of running kubelet. RedHat/CoreOS folks who initially championed this approach have given up on it, there are no test suites, no owners. sig-storage is having a hard time trying to keep things working. So please ping the CoreOS folks to research for an alternative for the future. Note that still runs kubelet in a container, it just does not use the flag or volume mounts as shown above.
Is it necessary to remove/deprecate the feature or just disable the flaky CI check? I find that feature and running kubelet in container helpful. Briefly looking at the implementation there is at least unit test suit.

that's the tip of the ice berg, there's a lot more things that can be simplified when we drop this feature, example see Another example of what we had to go through is here - There is no appetite to continue taking on this burden. I'll let and others chime in on the undue cost of keeping this alive.
fwiw I don't think we should deprecate the entire use case b/c it's highly valuable for testing. I'm not certain why this matters for the kubelet, b/c it really shouldn't unless it's a design flaw.
can you please chime in with the ongoing cost of supporting this feature? how much effort is to keep it going?
Yes we use this list of volume: This seems to work pretty nice for us and we don't have any volume issues so far.
Maintenance of containerized kubelet is increasingly painful. Kubelet / its storage code must know about all mounts on the host, resolve symlinks on the host, execute various helpers on the host and similar stuff, which is especially hard when it does not actually run on the host but in a weird container. See and for all the hacks we use. They are brittle and tend to break in very unexpected way. It's also largely undocumented and various ways how to run kubelet in a container are cargo-culted around. See above, , really? No offense, I know why it's there, but whole containerized kubelet is hack on top of hack on top of hack.
Not only is the code complex, but it is also not tested, so we have had to rely on manual testing when making bug fixes, and we don't know about regressions until after the release when users report issues. sig-storage has tried for 6+ months to find an owner that can dedicate resources to support and maintain this environment, but nobody has stepped up.
+1 to what asnd said. Either we should deprecate the feature, or find an owners who can commit to fixing issues and adding much needed tests. We tried to find an owner in SIG storage during Q2 2018 but no one stepped up.
Just because I like causing trouble, another alternative is to run executable from a container (or systemd portable unit, etc), but unify the kubelet's rootfs view with that of the host. Specifically: do the equivalent of , but for .

Then you would run from docker with (or equivalent), and early in chroot() to /rootfs. This loads the executable from the docker image, but all later filesystem operations are done from a pov equivalent to the host root fs. I think this keeps the kubelet lifecycle management benefits of running the kubelet binary from a container and allows the storage folks to just do regular filesystem operations, like there was nothing clever going on.
The function needs ownership from both SIG Node and SIG Storage. There were attempts early in the project to remove containerized kubelet support, but Red Hat offered contributors to maintain this model in both SIGs for ~11 releases of Kubernetes. Kubernetes was much smaller 11 releases ago and our shared project experience was limited. It's one thing for a feature to work initially, and another thing to reason about the impact of alternative deployment models when dealing with CVE responses or just doing normal maintainer review of incoming PRs or feature function. That lived experience is informing our future choices at Red Hat to focus on just deploying kubelet as systemd unit (like many other Kubernetes service providers), but if there are others in the community that want to keep the function alive and maintain it we need owners to volunteer and commit to fix issues and add much needed tests. If not, it seems the responsible thing for the community is to clearly deprecate it.
I like this suggestion to simplify the mechanism while retaining the packaging and runtime benefits of the container image. I'm curious what we're doing in talos if we're not using the flag and how what we're doing solves the use-case of running kubelet inside a container.
I kind of like the idea of mounting to as well. Although, one of the benefits we see in mounting only the needed host paths is that it is less of the host filesystem exposed in case of a security vulnerability. On the other hand we currently mount everything we think we need: and that could change in any release. The above solution would allow us to never worry about that again really. We don't use the flag, and I can't say I have seen any of the mentioned issues in our setup. We pass Sonobuoy conformance tests as well. So I can't say I would know exactly what we are doing to solve these problems since I personally have ran into them. I have had rook run just fine as well.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }, InitialDelaySeconds: 15, TimeoutSeconds:      5, <del> FailureThreshold:    1, </del> <ins> FailureThreshold:    5, // to accommodate nodes which are slow in bringing up containers. </ins> }, }, },

We have noticed few months ago that this test flaked again() Opening this issue again to start dicussion on the test. Following are my observations after going through history and logs : We are bringing up a nginx container and trying to check if <container_IP:80is available. While this is not a bad idea, there are many things that could go wrong because of which nginx may not come to running. Some of them could be related to underlying infrastructure on which we have no control over, some could be related to network plugin being used etc. Some were mentioned in kubernetes/kubernetes (comment) as well. For example, the latest one failed with dial tcp 10.128.1.69:80: getsockopt: no route to host. This could be related to network setup as well. This makes the test indeterministic and therefore flaky. Since the goal this test is to check that a good container won't restart based on liveness probe, would it make sense to add a liveness command that checks for the existence of a particular directory that was created during container start up or something else instead of a HTTP GET. /sig node cc
not sure if this is helpful, but to my understanding, this is technically already supported as described in this page. You just have to figure out which probe you need most, either httpGet or exec.
It is not about configuring which probe to use. The problem is with test case flaking out and it flakes for what I believe could be an infrastructure related issue. I raised this issue to ensure that everyone is ok switching to exec from httpGet.
- I just noticed that we already have a test which does exec liveness probe test. Do we still need the http test? If not, can we delete it or increase timeouts?
/kind flake


Question: // Deletes the passed-in pod and waits for the pod to be terminated. Resilient to the pod // not existing. func DeletePodWithWait(f *Framework, c clientset.Interface, pod *v1.Pod) error { <ins> const maxWait = 5 * time.Minute </ins> if pod == nil { return nil } <del> Logf("Deleting pod %v", pod.Name) </del> <ins> Logf("Deleting pod %q in namespace %q", pod.Name, pod.Namespace) </ins> err := c.CoreV1().Pods(pod.Namespace).Delete(pod.Name, nil) if err != nil { if apierrs.IsNotFound(err) { <del> return nil // assume pod was deleted already </del> <ins> return nil // assume pod was already deleted </ins> } <del> return fmt.Errorf("pod Get API error: %v", err) </del> <ins> return fmt.Errorf("pod Delete API error: %v", err) </ins> } <del> // wait for pod to terminate err = f.WaitForPodTerminated(pod.Name, "") if err != nil && !apierrs.IsNotFound(err) { return fmt.Errorf("error deleting pod %q: %v", pod.Name, err) } if apierrs.IsNotFound(err) { Logf("Ignore "not found" error above. Pod %q successfully deleted", pod.Name) </del> <ins> Logf("Wait up to %v for pod %q to be fully deleted", maxWait, pod.Name) err = f.WaitForPodNotFound(pod.Name, maxWait) if err != nil { return fmt.Errorf("pod %q was not deleted: %v", pod.Name, err) </ins> } return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: See One /sig storage
/assign
[Slow] test suite was run. Glusterfs io stream test failed due to ns not being deleted, which was due to 1 pod remaining. The glusterfs-server pod was deleted. It appears that the glusterfs-client pod was not deleted even though returned no error. In cases where this test succeeds (which is most of the time) returns which is seen in the log as INFO: Ignore "not found" error above. Pod "gluster-io-client" successfully deleted. Sometimes when the plugin io streaming tests pass you'll see logged for the client pod. However, sometimes a client pod is deleted but the above log entry is not triggered. Eg in you'll see the nfs-client pod is deleted (the ns was able to be deleted) yet the above log msg stating to ignore the "not found" error was not generated. (scroll down to the line "I0724 20:27:20.364] Jul 24 20:27:00.559: INFO: Waiting for pod nfs-io-client in namespace 'e2e-tests-volume-io-pmdks' status to be 'terminated due to deadline exceeded'(found phase: "Running", readiness: true) (29.883170178s elapsed) I0724 20:27:20.365] [AfterEach] [sig-storage] NFS"). My guess is that in this case the plugin's client pod did not terminate in 30s -- WHY NOT THE FULL 5m? -- but did terminate in time for the ns to be deleted. And sometimes the client pod cannot be deleted in 30s (again WHY NOT THE FULL 5m?) and is still present when the ns is deleted. Note: if we cannot detect that the client pod was not actually deleted and then we delete the server pod, there is no chance that the ns can be deleted. Once the server pod is gone the client pod will not delete.
It looks like the client pod failed to terminate:
30s may not always be enough time for the gluster client pod to terminate?

also, why is it so common to see ~29 seconds for the client pods (nfs, gluster) to supposedly terminate. The wait should be for up to 5m yet I never see = 30s wait times... yet, we know that sometimes the gluster client pod is not terminated. Seems like maybe a bug in this area?
can you take a look? My findings so far are this test fails about 20% of the time. The gluster client pod's Status is set to Failed and therefore the pod won't be deleted. There is e2e code that waits for the pod to be deleted but escapes if the pod has "Failed" and if the passed-in reason matches the pod's Status.Reason. In the gluster pod test failures the pod's Status.Reason == "". This causes the test to incorrectly assume that the pod was deleted when it wasn't. This problem manifests itself when the namespace is deleted and that step fails. It seems like the Failed pod's Status.Reason should not be blank. cc
It looks like gluster unmount failed:
But in the log I looked at today (https://k8s-) I see the pod's Status=Running. Then the pod is deleted and the status remains Running for ~30s then changes to Failed. Even though the pod is Failed its Reason remains "". So 1) I wonder why the pod is marked as Running if its mount failed, and 2) once the pod is Failed why is Reason still blank?
It was the unmount that failed, not the mount.
ah, ok thanks, I missed that. Still need to know why Pod.Status.Reason is blank...
Below are some records from a related to the glusterfs-client pod, gluster-io-cliente2e-tests-volume-io-6h140(-778e-11e7-afad-)_ and the "e2e-tests-volume-io-6h140" ns:
I think the issue is that the test is deleting the gluster endpoints before deleting the pod.
Nm, the test does seem to be cleaning up in the correct order: client pod, then server pod and endpoints. So the issue seems to be that during Pod termination, it goes to Failed state.

Before, I thought it was because the unmount failed, but based on the timestamps of the logs, it seems like it fails before we start unmounting.
Just spoke to and he said that it's normal for a Pod to go to Failed state during the termination sequence. In between Failed state and Pod object deletion is when we unmount the volumes. So I think the issue here is that the WaitForPodTerminated function needs to wait only for the Pod object to be deleted, and not return early if Pod state is Failed.
do you have time to look at this? The test is still flaking
yes I can take a look. Do you know if there has been a pr started to address this issue? BTW, I will be gone most of tomorrow but will start tomorrow night.


Question: Port *intstr.IntOrString `json:"port,omitempty" protobuf:"bytes,2,opt,name=port"` } <del> // IPBlock describes a particular CIDR (Ex. "192.168.1.1/24") that is allowed to the pods // matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should // not be included within this rule. </del> <ins> // IPBlock describes a particular CIDR (Ex. "192.168.1.1/24","2001:db9::/64") that is allowed // to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs // that should not be included within this rule. </ins> type IPBlock struct { // CIDR is a string representing the IP Block <del> // Valid examples are "192.168.1.1/24" </del> <ins> // Valid examples are "192.168.1.1/24" or "2001:db9::/64" </ins> CIDR string `json:"cidr" protobuf:"bytes,1,name=cidr"` // Except is a slice of CIDRs that should not be included within an IP Block <del> // Valid examples are "192.168.1.1/24" </del> <ins> // Valid examples are "192.168.1.1/24" or "2001:db9::/64" </ins> // Except values will be rejected if they are outside the CIDR range // +optional Except []string `json:"except,omitempty" protobuf:"bytes,2,rep,name=except"`

<!-- Please only use this template for submitting enhancement requests --What would you like to be : mentions an IPBlock with only IPv4 examples. Add a comment or example to clarify that IPv6 network blocks are (or possibly aren't) valid here. Why is this needed: Cluster operators who use IPv6 networking want to understand how to isolate different workloads inside their cluster network (and without reading source code). Logged against kubernetes/kubernetes rather than the website repo because this is generated documentation. Kubernetes already has single-stack support for IPv6 clusters KEP for dual-stack support is /sig docs /sig network /priority backlog
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign /remove-triage unresolved can you guide me on what steps are needed to publish the examples?
I via the pull request. (Although I want these examples , it's not a task I'm familiar with).


Question: // Input args must follow the format and sequence of ebtables list output. Otherwise, EnsureRule will always create // new rules and causing duplicates. EnsureRule(position RulePosition, table Table, chain Chain, args ...string) (bool, error) <ins> // DeleteRule checks if the specified rule is present and, if so, deletes it. DeleteRule(table Table, chain Chain, args ...string) error </ins> // EnsureChain checks if the specified chain is present and, if not, creates it.  If the rule existed, return true. EnsureChain(table Table, chain Chain) (bool, error) // DeleteChain deletes the specified chain.  If the chain did not exist, return error.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!


Question: import ( "bytes" "encoding/json" <ins> "fmt" </ins> "io/ioutil" "net/http" "os"

From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct.

Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now?

I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016


Question: # Pre-Condition: namepace does not exist and no POD exists output_message=$(! kubectl get namespace multi-resource-ns 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'namespaces "multi-resource-ns" not found' <del> kube::test::get_object_assert pods "{{range.items}}{{${id_field:?}}}:{{end}}" '' </del> <ins> kube::test::wait_object_assert pods "{{range.items}}{{${id_field:?}}}:{{end}}" '' </ins> # First pass, namespace is created, but pod is not (since namespace does not exist yet). output_message=$(! kubectl apply -f hack/testdata/multi-resource-1.yaml 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'namespaces "multi-resource-ns" not found'

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: runkubectlapplytests Testgrid link: Reason for failure: A test has a precondition that no pods should be up, but, previous test's deployment did not fully delete. As a result precondition check fails. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig testing
Pasting the failed tests' output, for easy reference:
Found one more flake test, due to same reason. Checking for more.


Question: if math.Mod(float64(fsize), float64(testpatterns.MinFileSize)) != 0 { fsize = fsize/testpatterns.MinFileSize + testpatterns.MinFileSize } <del> fpath := path.Join(dir, fmt.Sprintf("%s-%d", file, fsize)) </del> <ins> fpath := filepath.Join(mountPath, fmt.Sprintf("%s-%d", file, fsize)) defer func() { deleteFile(clientPod, fpath) }() </ins> if err = writeToFile(clientPod, fpath, ddInput, fsize); err != nil { return err } if err = verifyFile(clientPod, fpath, fsize, ddInput); err != nil { return err } <del> deleteFile(clientPod, fpath) </del> } return

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow] Since when has it been failing: 11/13 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: This is consistently failing in 1.13 branch and is release blocking. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test /sig storage /priority critical-urgent /milestone v1.13 /assign /cc
The test case is failing because it wrote 2MB instead of the expected 1MB. It's also very odd that the test is failing in soak, but passing in regular CI suites.
There's a few problems: .GetVolumeSource returns "/tmp" for everything. This means all tests will share the same directory. This should be probably use some unique directory instead volume IO test generates a unique mountPath directory, however that doesn't mean a unique directory is actually created on the host. Ie, a mountPath of "/a/b/c" on a hostpath volume at "/tmp" doesn't mean directory "/tmp/a/b/c" is created. io test file name is not unique seems like the IO test is not cleaning up the files it creates, so the files are getting reused on subsequent tests due to the non-unique naming issues.
The other thing I can't explain is why it's failing only now
Fix is here:
As per discussion in 1.13 burndown this morning, this is a test job issue and its not release blocking at this point. So we can take this fix in 1.14 as opposed now during code freeze let us know if you feel otherwise
/milestone v1.14 /priority important-soon
/remove-priority critical-urgent


Question: } for _, lb := range serviceLBNames { framework.Logf("cleaning gce resource for %s", lb) <del> framework.CleanupServiceGCEResources(lb) </del> <ins> framework.CleanupServiceGCEResources(lb, framework.TestContext.CloudConfig.Zone) </ins> } //reset serviceLBNames serviceLBNames = []string{}

In federation tests, if we observe the build logs. we see cleaning up LoadBalancer from cloud provider seems not working. It has 2 effects: delays the tests by waiting to delete but always returns error. so we waste like 5 minutes. maybe we are leaking the LoadBalancer in the project.
we can see these errors in every build. but just for reference here are some build logs https://k8s-
cc
All the errors in logs are . It doesn't necessarily indicate we are leaking resource, just now we must have clusterID initialized before calling EnsureLoadBalancerDeleted(). Would be better to not call this function directly in e2e test :)
Federation cleanup should use the same implementation as non-federated e2e test:
function looks like only specific to GCE. in future we would run the e2e tests in AWS too. it would be better to use generic functions if any.
using is working for us. here is the test logs https://k8s-
EnsureLoadBalancerResourcesDeleted() actually cleans up nothing, it only lists forwarding rules and probably logging :)


Question: if cfg, err := ReadDockerConfigFile(); err == nil { return cfg } else if !os.IsNotExist(err) { <del> glog.V(1).Infof("Unable to parse Docker config file: %v", err) </del> <ins> glog.V(4).Infof("Unable to parse Docker config file: %v", err) </ins> } return DockerConfig{} }

As per We should look for the .dockercfg in three places: ./.dockercfg, ~/.dockercfg, and $rootDir/.dockercfg
I'll tackle this.
Bunch of questions: we also add a flag that overrides the search mechanism suggested in this issue? Perhaps, ? to the user's homedir requires we use os/user [0] . The os/user package and cross-compiling don't play nice [1]. So, if we add to the search path, cross-compiled kubelet binaries will not actually be able to take advantage of it. Is this an issue? is the proper lookup order? I suggest: if provided, , , . walking through the lookups, do we stop at the first file that exists, or the first file that exists and is parsed as a valid ? [0] [1] /cc
I would absolutely love a --dockercfg option, and it would also be helpful if it logged which .dockercfg it ultimately used.
1) Seems like YAGNI to me 2) Is all of os/user broken? We will eventually want actual user support. I'm fine with dropping it for now. 3) I would say $rootDir/.dockercfg, ./.dockercfg (assuming we drop ~) 4) First successful parse sounds good to me. And can we NOT log errors if files don't exist :)
would it make sense to default to the kubelet config directory if provided rather than the working directory of the kubelet process? then no extra flag required.
What ever happened with this idea?
/cc
It never got high enough on my weekend TODO list :( I can keep telling myself that I need to do it, or we can find someone who has a bit more time than me.
I can probably get to this today.
Cool, thank you! , Mike Danese wrote:


Question: 1.17 will work on the current Aggregator/API-Server. release: v1.17, v1.21 file: test/e2e/apimachinery/aggregator.go <del> - testname: Aggregator, manage lifecycle of an APIService codename: '[sig-api-machinery] Aggregator should manage the lifecycle of an APIService [Conformance]' description: An APIService is created which MUST succeed. The APIService status when replaced MUST succeed. Given the updating of the APIService status, the fields MUST equal the new values. The APIService status when patched MUST succeed. Given the patching of the APIService status, the fields MUST equal the new values. The APIService when replaced MUST succeed. Given the updating of the APIService, the fields MUST equal the new values. It MUST succeed at deleting a collection of APIServices via a label selector. release: v1.25 file: test/e2e/apimachinery/aggregator.go </del> - testname: Custom Resource Definition Conversion Webhook, convert mixed version list codename: '[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert a non homogeneous list of CRs [Conformance]'

pull-kubernetes-e2e-kind [sig-storage] CSI .* It seems that happens since 7/8 The test timeout the docker daemon is killed, but the process keeps running, but it fails to do anything because all the features are missing, in this case, the port forwarding from the host to the apiserver container, as we can see in the following log entry As consequence, all commands fail, like the ones that gather the logs or clean up the environment The sig storage test take more than 10 minutes we shouldn't have test with that duration on presubmit, those also should be tagged as slow ~The trigger may be related to the ginkgo v2 update, the failures start to happen the same day that merged~ /sig testing /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc I know this people is in vacation, ccing just for visibility /cc
I don't know if correlation implies causality here, but it seems to start the day that the ginkgov2 was merged /cc is there any change in behavior for which the tests keep running, I remember some discussion about this between you and about the failure modes
hey there correlation doesn't necessarily imply causation - but it's definitely the first place to start looking ;) there shouldn't be any explicit changes in V2's behavior here however there were changes around how Ginkgo captures stdout/stderr during parallel test runs (i.e. when running with ) that, in some circumstances involving external processes that hang, can cause to hang. Often when this happens Ginkgo catches it and prints out a message but I assume you haven't seen anything like that? Here's something to try: If: (a) these specs are running in parallel (b) the failure is somewhat reproducible Then: try running with to see if the flaky failure is resolved. If the test consistently exits without hanging with this flag on then that would point to this output-interception issue and potentially help us figure out the next step for debugging things.

This sort of failure would also happen in Ginkgo v1 but it would be intermittent. V2 has a much cleaner implementation of output interception which has the side-effect of making this sort of failure more consistent. usually the issue ends up being with a spawned process holding on to an open pipe or failing to clean up somehow.... it can be a gnarly thing to debug :(
it seems that the CSI tests are taking into account the time to delete the namespace, per example takes 10 mins, starts at and takes 5 mins deleting the namespace it feels like this is related on how CSI tests are structured, made some comments about this here it seems that is the problem, the sig-storage test take the time to delete namespace into account , maybe for the way the use the hooks for AfterEach, can this be refactored, the current CI presubmits are very flake because of this problem
we are at a success 47% rate on presubmits, /priority critical-urgent
Hi I'm not sure I understand the issue. Is there a change in cleanup behavior to cause these tests to start timing out? cc who also looked into storage test cleanup in the past.
Yeah, it seems to be related to the ginkgo v2 migration, there were some behaviour changes ... I'm just trying to get some help because I couldn't find the problem and I am lost on those CSI e2e test logic
i put this in the sig-testing slack but in case it's relevant here: one thing to note is that in ginkgo v1 the —timeout flag used to apply to each suite separately. So if you are running 3 suites with a timeout of 50 minutes each suite gets 50 minutes. In v2 the timeout governs the entire test run - that means all 3 suites must finish within 50 minutes. not sure if that is in play here (are there multiple suites running?)
let's try this theory
can you please take on the slow tests from sig-storage that are running on presubmits?

I can do it but I'm afraid of breaking something, you can see them here
This seems to have improved
nope, it didn't improve, it may be an artifact of not having many PR or low load on the CI, who knows
hey -- looking at just that last run I'm seeing: 13:18 is the first timestamp I see. Eventually (~around 13:35) the specs start running. Everything is happy and several hundred specs run in parallel and pass. The last three specs then fail - these three are running in parallel. The first spec is: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand [AfterEach] should resize volume when PVC is edited while pod is using it and has an error trying to delete the test namespace: The seems to wrap it in a try and so this timeout/failure doesn't mark the spec as failed. The storage specific cleanup keeps running... This stuff: appears to succeed - but then this step: seems to hang. Eventually, at 14:18:32 I see: Presumably after three minutes of polling the namespace hasn't been removed. But now that the outer container/infrastructure has timed out the tcp connection attempting to continue polling fails. It looks like that 1 hour timeout is governed by the test setup, nto the Ginkgo test suite. With the container/infrasturcture gone, the spec reports the error "". but this too is wrapped in a try. The spec finally fails in the with: again, because the container/infrasturcture was killed at 14:18:32. This failure now bubbles up as the failure associated with the spec. But in reality the underlying issue seems to be a failure to delete the namespace. The second spec is: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral [AfterEach] should create read-only inline ephemeral volume Here too we see: ^ this appears to be the actual failure. Eventually the reports a failure at 14:18 after the container/infrastructure times out.

Finally, the third spec is: [sig-storage] CSI mock volume [AfterEach] Delegate FSGroup to CSI driver [LinuxOnly] should not pass FSGroup to CSI driver if it is set in pod and driver supports VOLUMEMOUNTGROUP here, once again, we see ^ this appears to be the actual failure. As with the other specs, eventually the reports a failure at 14:18 after the container/infrastructure times out. So, it looks like something_ happens around 14:08 that puts the cluster in a bad state and these volume-related namespaces time out trying to delete. This pushes the run time for the entire suite (including the setup bits before Ginkgo runs) past the 1 hour limit imposed by the container/infrastructure layer. I'm not sure this is a V1 vs V2 issue (perhaps some previously implicit behavior aroudn test ordering has changed - but these parallel specs basically run the same way in V1 as in V2). Have there been any changes to the storage stuff introduced around the same time as the V2 upgrade? Perhaps this is an infrastructure resource/exhaustion issue - 700 specs managed to succeed before these last three failed!
I don't know if the custom cleanup and the aftereach are conflicting and hanging
no, i don't believe so. they are running sequentially for a given spec.
we do some crazy things, :smile: We need to minimize the diff to find out the root cause, and we can't have the presubmit jobs with a 45% success rate, I've reverted the AfterEach change and I'm asking sig-storage to tag the slow tests accordenly to current policies (5 mins) The lead now is in:
I am attempting to do some bisection... Here's a job from 7/5/22 where all of the csi tests are finishing in under 3 minutes:
Here's a later run with quick running times: I think I need to be looking at periodics though, not pull jobs
here's the earliest run I found with slow times:
Looking at CI jobs: Here's the first job where tests started taking 10 mins: The run before that was fine:
nicely done ! how does one figure out which commit has those two runs correspond to?

Good run was based on the following commit: Slow run was on the following commit: Not a lot of commits between the two and none of them look particularly suspicious
Towards the beginning of the test log, it shows the kubernetes version used to create the kind cluster:
any changes merge to test-infra around the range between those runs (and maybe slightly before)?
not really:
kind versions also look the same
BTW here's the prow job history page starting on the last good run: There is a very clear step up in test time after that.
I think next step is to try to locally rerun the tests against the good commit and the two commits after the good run to see if we can rule them out.
Cross posting from find the report attached with the logs for successful tests I've checked Test starts It seems the problem is on the namespace deletion The storage tests add a custom hook to delete the namespace that seems that is the one causing they take longer
The storage tests run all these cleanups inside the test context has namespace deletion changed somehow ? any possible idea on how ginkgov2 could affect this workflow?
I run this test repeatedly on my desktop, the test passed every time with around 50 seconds to finish the test. And I did some random and destructive testing - terminate the test execution in the middle of process, and hit the situation that namespace deletion is hanging, I check the log and found that is because there is a pod in that namespace, and both the namespace and pod are hanging at the phase of . And the termination never finish unless I force to remove them. Here is the log I can see from kubelet, show that has already been created. I suspect the hang we saw should hit the situation similarly, something in that namespace, pod? prevent us to delete the namespaces successfully. Current code doesn't dump the resource in that namespace since the namespace deletion is done in the , here is the code for that. So, is that possible we could hit the similar situation when all of the testcases run together? we could be able to dump the everything in that namespace when the termination is hanging. will dump the logs from each components, such as controller manger which might give us some hints on this issue, but I don't know where to get those logs.
The reason why the CSI tests manage namespace deletion is because of deletion ordering issues.

Each test installs its own CSI driver, however that CSI driver has to be deleted last otherwise Pods will fail to teardown and PVCs will fail to delete due to an uninstalled CSI driver.
make sense, but the log says the is not found. but I do find that csi driver is presented in that cluster, what am I missing here?
the and stuff in these tests is complex and i do wonder why the code isn't calling and instead. nonetheless i would expect this code as written to work: the will always run before the returns to its caller and Ginkgo v1 and v2 both basically guarantee that for a given process no two s will run in parallel - so all those closure variables shouldn't see any pollution from one test to the other within a given process. but. there's a lot of code here and while i don't see any obvious places where test pollution is creeping in due to these shared variables i can't say i'm 100% confident. also - while i don't yet see how ginkgo v2 vs v1 could be causing this we definitely haven't ruled anything out yet! one question: where are these namespace integers being generated (eg ) are we confident that the other parallel processes will never also pick 2296? or is it possible that multiple processes are picking the same integer and so cross-process pollution is occurring? it could be a good use of time run the suite while focusing just these CSI specs and running them in parallel with verbosity turned on so we can see what's going on. my sense is the current hypothesis is that a passing test is inadvertently deleting a resources that another test running in parallel is using thereby causing it to run slower. since we don't run with in verbose mode we won't see the relevant logs from the passing test. that would be my suggestion for a next step. run just these CSI specs in parallel with and see if we learn more from the logs.
I think I found the root cause. I did some debugging on my own PR, and debugged the GCE run because the kind run does not capture kube-controller-manager logs: test log: kube-controller- The commit that I thought wasn't suspicious is actually suspicious:
cc please help take a look. I think the new tests are disruptive and causing namespace deletion in other tests to fail.
Great to hear that!

Can you pls share us how to achieve that if we want to check the log from those components? where is the log file I can get?
For example from my test run above, this is where the control plane logs are: It would be nice to collect this from the kind jobs too.
~ (sorry) :) It's historical reason, my pr here: already commented on that, further actions are needed, this is also the action left in We just need time to get familiar with csi e2e test code and then start the migration. It's generated here: , and used here: In theory, I think this is possible, but the how the probability could be. Good suggestion, I think we can add this parameter along with the to the test code, and make it masked with comment. Anyway, thanks for the root causing!
the info is there, but the failing jobs doesn't have it because the job times out cleaning the kind cluster and the related logs If you take a working job the controller-manager logs are in and this confirms theory and demonstrates once more that correlation doesn't imply causality my fault :(, sorry all for the noise
another observation that, once this test started to run, it caused the job to flake !
great job getting to the bottom of this !
no failures since the offending test was reverted ! confirming msau theory
that is itself a bug, and shouldn't be happening. we should get and handle a TERM from prow well before we get killed and the TERM handler should be capturing logs.
log collection in timed out kind e2e runs should be fixed now , , sorry about that


Question: } func NewResourceExpirationEvaluator(currentVersion apimachineryversion.Info) (ResourceExpirationEvaluator, error) { <del> ret := &resourceExpirationEvaluator{ // TODO https://github.com/kubernetes/kubernetes/issues/109799 set this back to false after beta is tagged. strictRemovedHandlingInAlpha: true, } </del> <ins> ret := &resourceExpirationEvaluator{} </ins> if len(currentVersion.Major) > 0 { currentMajor64, err := strconv.ParseInt(currentVersion.Major, 10, 32) if err != nil {

once the we hit a beta release, we can reset the default value for strict handling to false to prep for 1.25. It will likely be introduced in /kind bug /priority important-soon /milestone v1.25 ignore template No response this time <details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
Hello my name is Justin Chizer and I am one of the k8s 1.25 bug triage shadow assigned to track this body of work. Is this still on track for 1.25?
This needs to be done now that master is on 1.26 until 1.25.0 is cut, since verify jobs are failing on removal of 1.26 deprecated APIs /priority critical-urgent


Question: } if !f.readOnly { <del> volume.SetVolumeOwnership(f, fsGroup) </del> <ins> if f.plugin.capabilities.FSGroup { volume.SetVolumeOwnership(f, fsGroup) } </ins> } return nil

What happened: For some filesystems recursively applying an SELinux label through chcon or the GID permission is expensive (30 mins in some cases). There is better strategies for the recursive change and the call should be available through FLEX API. What you expected to happen: FLEX has a shell script call to label or set GID for a volume. Notes-- The fix should be backwards compatible.. If the shell call to perform the permission change is absent, then we fall back on the original method.