Question: framework.ExpectEqual(*(rs.Spec.Replicas), rsPatchReplicas, "replicaset should have 3 replicas") framework.ExpectEqual(rs.Spec.Template.Spec.Containers[0].Image, rsPatchImage, "replicaset not using rsPatchImage. Is using %v", rs.Spec.Template.Spec.Containers[0].Image) } <ins> // List and DeleteCollection operations func listRSDeleteCollection(f *framework.Framework) { ns := f.Namespace.Name c := f.ClientSet rsClient := f.ClientSet.AppsV1().ReplicaSets(ns) one := int64(1) rsName := "test-rs" replicas := int32(3) e2eValue := rand.String(5) // Define ReplicaSet Labels rsPodLabels := map[string]string{ "name": "sample-pod", "pod":  WebserverImageName, "e2e":  e2eValue, } ginkgo.By("Create a ReplicaSet") rs := newRS(rsName, replicas, rsPodLabels, WebserverImageName, WebserverImage, nil) _, err := rsClient.Create(context.TODO(), rs, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Verify that the required pods have come up") err = e2epod.VerifyPodsRunning(c, ns, "sample-pod", false, replicas) framework.ExpectNoError(err, "Failed to create pods: %s", err) r, err := rsClient.Get(context.TODO(), rsName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get ReplicaSets") framework.Logf("Replica Status: %+v", r.Status) ginkgo.By("Listing all ReplicaSets") rsList, err := c.AppsV1().ReplicaSets("").List(context.TODO(), metav1.ListOptions{LabelSelector: "e2e=" + e2eValue}) framework.ExpectNoError(err, "failed to list ReplicaSets") framework.ExpectEqual(len(rsList.Items), 1, "filtered list wasn't found") ginkgo.By("DeleteCollection of the ReplicaSets") err = rsClient.DeleteCollection(context.TODO(), metav1.DeleteOptions{GracePeriodSeconds: &one}, metav1.ListOptions{LabelSelector: "e2e=" + e2eValue}) framework.ExpectNoError(err, "failed to delete ReplicaSets") ginkgo.By("After DeleteCollection verify that ReplicaSets have been deleted") rsList, err = c.AppsV1().ReplicaSets("").List(context.TODO(), metav1.ListOptions{LabelSelector: "e2e=" + e2eValue}) framework.ExpectNoError(err, "failed to list ReplicaSets") framework.ExpectEqual(len(rsList.Items), 0, "filtered list should have no replicas") } </ins>

[X] test approval issue : [x] test pr : [x] two weeks soak end date : 2 June 2021 [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints which are untested. Create a ReplicaSet List all Replicasets DeleteCollection of ReplicaSets Varify the ReplicaSets have been deleted Using an existing conformance test as a template for a which validates that two new endpoints are hit. example endpoint | useragent --------------------------------------------|
createAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets deleteAppsV1CollectionNamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets listAppsV1ReplicaSetForAllNamespaces | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets readAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets (4 rows) If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
/triage accepted For Release 1.22
/assign
Dear When I run It only returns not listAppsV1ReplicaSetForAllNamespaces. What does this mean? What should I do? Regarding What should I see after implementing the tests? All 4 rows? Right now I see 0 rows when running it (without editing code) Lastly, could you tell me which parts need to be addressed in the ? Right now, it looks like the template does exactly what you want to write for this test.
This issue is created to show a planned PR to the conformance meeting. It is just a proof of concept before creating the conformance PR. What we do: I create a local cluster and create a test. In the cluster, I have running to see what user agents are hit by the test. Before I run the test I delete all cluster logs.

Then I run the test. This is my test: Once the test pass I run the APISnoop query to see which user-agents are hit. I in this case 4 API's are hit, but I am only interested in and as they are the ones yet.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: for _, mountPoint := range mountPoints { const sysfsDevice = "sysfs" <del> if mountPoint.Device != sysfsDevice { </del> <ins> if mountPoint.Type != sysfsDevice { </ins> continue } // Check whether sysfs is 'rw'

On kubernetes 1.4.6 taken from releases. Using CentOS 7.2 Kubernetes worked with a 1.2 version. I just uninstalled and redo installation with 1.4.6. SELinux is disabled Everyhting seems to works (controller-manager, kubelet, and so on...) excepting kube-proxy that returns an error: "sysfs not mounted". API is responding (internally and externally), and I can see nodes The command that is launched by systemd is (--v=8 to get a maximum of information): But sysfs is mounted on /sys: I also tried to set /sys as writable (only to be sure, with chmod +x), and also tried to use userspace proxy mode. Note: 1.2.6 version was ok I think that the problem is that sysfs mount "none" as label, not "sysfs".
Note: I can now confirm that the problem comes from the "label". I did this: Then kube-proxy successfully started.
before setting . This logic was introduced in v1.3 by to workaround a known docker issue That's why "1.2.6 version was ok". Because sysfs is mounted as , kubeproxy couldn't find it then reported error. But why sysfs is mounted as ?
I installed kubernetes on a "scaleway" (a very nice cloud provider) VPS cluster. Their VPS uses a init file that mounts fs like that. I opened an issue on their github. But anyway you may mount "none" for tmpfs or any ramfs, sysfs, bugfs. What I think is that the mount listing function that is used in kube-proxy code tries to list the mount "source" but not the "type". For that kind of mountpoint you can use "none" as source.
the problem is here: You should check .Type and not .Device. "Device" is what I named "label" or "source" because I didn't remember the name ;) Sysfs can mount a "none" device as type "sysfs", so maybe you can change. See:
I'll do a PR
SGTM.


Question: return false } <del> if _, hasExcludeBalancerLabel := node.Labels[LabelNodeRoleExcludeBalancer]; hasExcludeBalancerLabel { return false </del> <ins> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.ServiceNodeExclusion) { if _, hasExcludeBalancerLabel := node.Labels[LabelNodeRoleExcludeBalancer]; hasExcludeBalancerLabel { return false } </ins> } // If we have no info, don't accept

For various reasons (network topology, heterogenous nodes, etc) a user may want to control the addition of a node to a cloud load balancer. A specific issue with the Azure Container Instances connector which introduces a virtual node can be found here: The right way to do this is via a node annotation, e.g.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen This is still a problem.. any word on when feature gate goes out of alpha?
Setup the enhancement tracking issue for this Still discussing with sig-network on how we want to move forward with this.


Question: panic("Not implemented") } <ins> func (elb *FakeELB) AddTags(input *elb.AddTagsInput) (*elb.AddTagsOutput, error) { panic("Not implemented") } </ins> func (elb *FakeELB) RegisterInstancesWithLoadBalancer(*elb.RegisterInstancesWithLoadBalancerInput) (*elb.RegisterInstancesWithLoadBalancerOutput, error) { panic("Not implemented") }

BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.


Question: })) // Check for kubelet event about the pod. ginkgo.By("checking for kubelet event about the pod") <del> framework.ExpectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { </del> <ins> framework.ExpectNoError(wait.Poll(framework.Poll, 5*time.Minute, func() (bool, error) { </ins> selector := fields.Set{ "involvedObject.uid":       string(podWithUID.UID), "involvedObject.kind":      "Pod",

Appears related to I don't see this flaking in the but it appears to be quite flaky in presubmit CI, where this is the only test on the job that fails. /sig testing node /kind flake /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/cc
/remove-kind failing-test
I was debugging this test this morning the apiserver logs shows the kubelet events come too late, the test times out, is like they are queued in the kubelet the test fails at 10:24:22 but the event arrives at 10:24:27 but kubelet times doesn't add up, there are vents generated by kubelet at 10:23:07 if we check another job , where the test passes, there is only a few seconds delay in a working job I've analyzed the number of kubelet events on a job with the test passing against failing, and the results shows that the failing test have similar number of events, except the ones with the Reason FailedMount
I could've sworn we'd like banned conformance tests from depending on event delivery because it's not guaranteed to be reliable?
right now the test waits 60 seconds for the event since events are not synchronous, should we have a more reasonable period? EDIT saw Ben's comment after I commented, anyway, the test is failing because it takes more than 1 minutes, at least I couldn't find any failure related to no have any event at all
I'm not familiar with this test, but you're right that Events and Reason aren't things that a Conformance test can depend upon Conformance tests are intended to be stable and backwards compatible according to the standard API deprecation policies. Therefore any test that relies on specific output that is not subject to the deprecation policy cannot be promoted to conformance. Examples of tests which are not eligible to conformance: anything that checks specific Events are generated, as we make no guarantees about the contents of events, nor their deliveryanything that checks optional Condition fields, such as Reason or Message, as these may change over time (however it is reasonable to verify these fields exist or are non-empty)
/milestone v1.23
Did some digging.

It looks like this has been tagged as Conformance since at least 1.2, ref: (sometime prior to 2015-10-07) So this appears to be a case where we missed going back through the existing corpus of conformance tests when we wrote the above doc. is where the rule originated (merged 2018-10-08) I think this test should be demoted from Conformance, since the very nature of what it tests is certain events being generated SIG Node is free to continue to treat this as release-blocking, but it's not something we can reasonable expect of any Kubernetes cluster
Code itself for corresponding test says 1.9: Git history shows the test basically hasn't changed in 1y:
I'll split out demotion into a separate PR, don't want to tangle this issue up
The first single instance of a timeout failure I see is on I'm going to see if I can try to bisect from there, probably merged recently.
to be clear: the test has been around and we should look at fixing it, I was just surprised to see the test name with , we should open a different thread/issue on that topic I am working on the kubernetes deb/RPM releases for the day or else I'd be taking a look.
,%20master%20(dev)%20[non-serial]&width=5 (IPv6),%20master%20(dev)%20%5Bnon-serial%5D&width=5
Opened to cover the conformance demotion
/priority critical-urgent I am proceeding as though we have a perf regression with kubelet events until proven otherwise. Others feel free to also TAL.
I can't reproduce locally on its own, I either will need to stress test it in parallel with some other stuff or I need a slower machine.
/reopen I don't want to just close this because the timeout was bumped if there's a possible regression
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Staring at notes: Looking at the journal logs I'm seeing some (mild, not unexpected) client-side throttling: So that looks to me like the kubelet thinks it successfully got the event out by 10:23:18 which is well within our 1m timeout. The timestamps in Antonio's post only look at POST, let me see if I can find the PATCH logs from apiserver
From That PATCH was received when the kubelet sent it so the event should have been updated and available... could there be an apiserver latency issue? /sig api-machinery
Possible repro on a just-rebased PR:
I see another PATCH at but I think that's just from the pod getting torn down, it matches the Kubelet logs
this is patching the pod status, probably setting the IPs or something like that, check the URI format is a patch on the pod objetct AFAIK those POST are creating the events , the URI format /api/v1/namespace/<namespace/events ,
latest failure 1:45 am CET fix merged 1:55 AM CET, let's keep it open one day more
I really don't think waiting from 1 -5m is an acceptable fix. This may be catching a performance regression. It has stopped the bleeding but I think the problem remains...
the events , by design, may take 6 mins , so it is completely legit that an event can take 5 mins, you cant' fail if take less (actually, by definition they are unreliable so it should be fine also to not receive it) Performance can not be tested this way, there has to be an explicit test for that, and that test can not use events of course
If the prior "median event lifecycle time" (in the context of this test) was previously < 1 min (and thus test flakes were rare) and now the "median event lifecycle time" has increased so as to trigger regular test flakes, that does seem like something interesting and worthy of prioritization. It just seems probable that there's some "there" there. This could also be explained by environmental changes, did anything change in the underlying test infra that correlates with the increased test failure rate?

We previously had issues where presubmits were failing when periodics were not, see
If the test was newly introduced and was flaking, I'd agree with you, but we've had this test at 100% green since like the 1.2 release, and it only started flaking in the past 48h. This suggests to me that we merged a regression, but since it landed during code freeze, it's very difficult to determine which of the dozens of PRs it was.
Fairly confident there have not been substantial changes to the infra in the past week. For blocking presubmits the infra is moved to public infra owned by the CNCF / SIG K8s infra and the configs are all in shows the CI code being updated regularly, but not the build/test cluster that executes against. KIND upgraded containerd/runc two weeks ago, but that doesn't fit. Given the timing it seems more like:
indeed, check my analysis, a failing job has 429 FailedMount events, a job passing the test only 277, the rest of the events are the same Is not the first time that we have a performance regression or an env problem that causes similar effects, I remember one because of runc, one issue with the DNS on CI machines (that elana linked), other with golang, ... but this is not uncommon at the end of the release when CIs are busier I don't disagree that there can be something, actually I totally agree, I disagree on using this test as a proof or measure of this "something", because: test verifies that, once you create a pod an event from the scheduler and from the kubelet is created. , looking at the code, can take 6 mins and by definition are not realiable , Tests should be clear about what they test , for this cases we need to define a threshold, in sig network we have a test for latency of services, maybe adding one for latency of pods? we can define a threshold there, sig-scalability has it
Latest in triage: It does appear that we had a bunch of CI flakes start showing up as well before increasing the timeouts in the test, so maybe not just a presubmit thing.
/assign
No, it hasn't failed since the timeout bump merged, but the bump suppressing the symptom and hence is making this much more difficult to root cause.

I'd suggest looking at some runs before the timeout was bumped?
Per I'm wondering if some of the timing regressions we're seeing could be due to a perf regression having switched to CRI v1 with containerd. I can't find any corresponding failures on CRI-O. /remove-priority critical-urgent since test is no longer flaking
/unassign I'm not actively working this one
I think this is similar to in that mainly dockershim jobs are affected.
FTR the jobs in the OP are not dockershim based, kubeadm's kinder jobs do often use dockershim but kind (not kinder) does not support dockershim. is looking good though, fwiw
can we close this now?
single 1.21 failure. i'm going to call this closed for now.


Question: objects = append(objects, template, pod) b.create(ctx, objects...) <del> // There's no way to be sure that the scheduler has checked the pod. // But if we sleep for a short while, it's likely and if there are any // bugs that prevent the scheduler from handling updates of the class, // those bugs should show up as test flakes. time.Sleep(time.Second) </del> <ins> framework.ExpectNoError(e2epod.WaitForPodNameUnschedulableInNamespace(ctx, f.ClientSet, pod.Name, pod.Namespace)) </ins> // Unblock the pod. class.SuitableNodes = nil

In Aldo pointed out that: Let's use that instead of sleeping. /sig node /triage accepted /priority long-term More robust test.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### What would you like to be ? Aldo pointed out that: There is. There should be a Schedulable condition set to false in the PodStatus. /triage accepted /priority long-term ### Why is this needed? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: storage, _ := newStorage(t) emptyName := validController emptyName.Name = "" <del> emptySelector := validController emptySelector.Spec.Selector = map[string]string{} failureCases := []api.ReplicationController{emptyName, emptySelector} </del> <ins> failureCases := []api.ReplicationController{emptyName} </ins> for _, failureCase := range failureCases { c, err := storage.Create(ctx, &failureCase) if c != nil {

It doesn't make sense to define pod template with a set of labels that is strict subset of replicaSelector (or at least I can't find an use-case). For that reason I think that the labels in pod template part of replication controller should be optional and user should care about them only if he wants to define a superset of what replicaSelector already defines.
I've heard the same comments from a client. I think that it'd be more intuitive if we always specify pod labels in the template (so that we can copy&paste pod definition), and replicaSelector is optional (by default pod labels).
It would be reasonable for the selector to be a subset. I'm fine with intelligent defaulting but we need to be careful not to conflate the 2 concepts, which I think would increase the amount of confusion.
Setting RC selector from pod template labels SGTM. Any concerns It wouldn't have to work with separate templates, though it would be desirable if it did. It would be nice if we could add preconditions based on other objects (resource version of pod template in this case) to etcd transactions.
I think we agreed in another issue that defaulting the selector from the pod template labels is reasonable. No argument from me. One problem with pod templates is that we lose the strict relationship between selector and template, so if you change the template the RC needs to behave intelligently (I refuse to create more pods because the template selector doesn't match mine, etc)
We should do this, and we should default replicas to 1. We could also consider defaulting replication controller labels from pod template labels. That way, essentially the only thing that would need to be specified for a singleton RC would be the pod template.
I don't think we can default replicas to 1, since we can't tell the difference between an explicit zero and a default zero, and I think there are lots of reasonable cases where you want to set replicas == 0 on create. (esp. in the context of socket activation)
Responded on the PR: We can distinguish default from 0 if we make it a pointer.


Question: // TestInvocationWithoutRuntime invokes the plugin without a runtime. // This is how kubenet is invoked from the cri. func TestTearDownWithoutRuntime(t *testing.T) { <del> fhost := nettest.NewFakeHost(nil) fhost.Legacy = false fhost.Runtime = nil mockcni := &mock_cni.MockCNI{} fexec := &exec.FakeExec{ CommandScript: []exec.FakeCommandAction{}, LookPathFunc: func(file string) (string, error) { return fmt.Sprintf("/fake-bin/%s", file), nil </del> <ins> testCases := []struct { podCIDR         string ip              string expectedGateway string }{ { podCIDR:         "10.0.0.1/24", ip:              "10.0.0.1", expectedGateway: "10.0.0.1", }, { podCIDR:         "2001:beef::1/48", ip:              "2001:beef::1", expectedGateway: "2001:beef::1", </ins> }, } <ins> for _, tc := range testCases { fhost := nettest.NewFakeHost(nil) fhost.Legacy = false fhost.Runtime = nil mockcni := &mock_cni.MockCNI{} fexec := &exec.FakeExec{ CommandScript: []exec.FakeCommandAction{}, LookPathFunc: func(file string) (string, error) { return fmt.Sprintf("/fake-bin/%s", file), nil }, } </ins> <del> kubenet := newFakeKubenetPlugin(map[kubecontainer.ContainerID]string{}, fexec, fhost) kubenet.cniConfig = mockcni kubenet.iptables = ipttest.NewFake() </del> <ins> kubenet := newFakeKubenetPlugin(map[kubecontainer.ContainerID]string{}, fexec, fhost) kubenet.cniConfig = mockcni kubenet.iptables = ipttest.NewFake() </ins> <del> details := make(map[string]interface{}) details[network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE_DETAIL_CIDR] = "10.0.0.1/24" kubenet.Event(network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE, details) </del> <ins> details := make(map[string]interface{}) details[network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE_DETAIL_CIDR] = tc.podCIDR kubenet.Event(network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE, details) </ins> <del> existingContainerID := kubecontainer.BuildContainerID("docker", "123") kubenet.podIPs[existingContainerID] = "10.0.0.1" </del> <ins> if kubenet.gateway.String() != tc.expectedGateway { t.Errorf("generated gateway: %q, expecting: %q", kubenet.gateway.String(), tc.expectedGateway) } if kubenet.podCidr != tc.podCIDR { t.Errorf("generated podCidr: %q, expecting: %q", kubenet.podCidr, tc.podCIDR) } existingContainerID := kubecontainer.BuildContainerID("docker", "123") kubenet.podIPs[existingContainerID] = tc.ip </ins> <del> mockcni.On("DelNetwork", mock.AnythingOfType("*libcni.NetworkConfig"), mock.AnythingOfType("*libcni.RuntimeConf")).Return(nil) </del> <ins> mockcni.On("DelNetwork", mock.AnythingOfType("*libcni.NetworkConfig"), mock.AnythingOfType("*libcni.RuntimeConf")).Return(nil) </ins> <del> if err := kubenet.TearDownPod("namespace", "name", existingContainerID); err != nil { t.Fatalf("Unexpected error in TearDownPod: %v", err) </del> <ins> if err := kubenet.TearDownPod("namespace", "name", existingContainerID); err != nil { t.Fatalf("Unexpected error in TearDownPod: %v", err) } // Assert that the CNI DelNetwork made it through and we didn't crash // without a runtime. mockcni.AssertExpectations(t) </ins> } <del> // Assert that the CNI DelNetwork made it through and we didn't crash // without a runtime. mockcni.AssertExpectations(t) </del> } //TODO: add unit test for each implementation of network plugin interface

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: Using Kubenet with IPv6 causes a panic panic: runtime error: index out of range goroutine 625 [running]: (kubenetNetworkPlugin).Event(0xc4203bf440, 0x462ebd1, 0xf, 0xc4219deae0) +0x889 This is due to IPv4 specific code in What you expected to happen: Kubenet should support IPv6 How to reproduce it (as minimally and precisely as possible): Running local-up-, use kubenet, and set the parameters for kubenet with ipv6 nodecidrargs="--allocate-node-cidrs=true --cluster-cidr=2001:beef::/48 --node-cidr-mask-size=64 " Anything else we need to know?: Environment: Kubernetes version (use ): latest as of June 26, 2017 Cloud provider or hardware configuration*: local on a VM OS (e.g. from /etc/os-release): CentOS Kernel (e.g. ): Linux localhost.localdomain 3.10.0-514.el7.x8664 SMP Tue Nov 22 16:42:41 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/area ipv6
/sig network /remove-sig api-machinery
/assign


Question: gvr("storage.k8s.io", "v1beta1", "csistoragecapacities"): { Stub:             `{"metadata": {"name": "csc-12345-2"}, "storageClassName": "sc1"}`, ExpectedEtcdPath: "/registry/csistoragecapacities/" + namespace + "/csc-12345-2", <ins> ExpectedGVK:      gvkP("storage.k8s.io", "v1", "CSIStorageCapacity"), </ins> }, // --

The override in can be removed in Kubernetes v1.25 because then the oldest supported kube-apiserver for downgrades will be v1.24 which will understand the v1 format. Also need to adapt See and Code cleanup...
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /lifecycle frozen


Question: cmd := fmt.Sprintf(`curl -s -o /dev/null -w "%%{http_code}" --connect-timeout 5 http://%s/healthz`, healthCheckNodePortAddr) out, err := e2eoutput.RunHostCmd(pausePod0.Namespace, pausePod0.Name, cmd) if err != nil { <del> return false, err </del> <ins> framework.Logf("unexpected error trying to connect to nodeport %d : %v", healthCheckNodePortAddr, err) return false, nil </ins> } expectedOut := "503" if out != expectedOut { <ins> framework.Logf("expected output: %s , got %s", expectedOut, out) </ins> return false, nil } return true, nil

pull-kubernetes-e2e-gce, assorted GCE jobs Seems to be elevated today No response No response_ /sig network
three instances:
/assign I've seen it too
/triage accepted
confirmed that it has been fixed !


Question: containers: - image: gcr.io/google_containers/heapster_influxdb:v0.3 name: influxdb <ins> ports: - containerPort: 8083 hostPort: 8083 - containerPort: 8086 hostPort: 8086 </ins> - name: grafana image: gcr.io/google_containers/heapster_grafana:v0.6 env:

/sig gcp /sig cluster-life-cycle /kind flake /milestone v1.10 cc https://k8s- starting last night /assign can one of you verify?
/kind bug /priority important-soon
/priority critical-urgent /priority failing-test /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/assign according to , gonna move those jobs to us-central1-c instead


Question: "testing" "time" <ins> "github.com/google/go-cmp/cmp" </ins> v1 "k8s.io/api/core/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/types"

Events from node taint manager doesn't report involved object apiVersion: ! apiVersion is reported as other events. For example: ! a pod on node A (not master) off node A is reported, but without apiVersion in involvedObject. The apiVersion is not reported in The issue is still not fixed in master: ! <details</details<detailsIrrelevant </details<details</details<details</details<details</details<details</details
/sig node /sig apps
/assign In the latest version, the issue is confirmed
one suggestion you should wait for the corresponding SIG to triage the issue and accept it before sending PRs, the chances of your PR getting reviewed and approver are much higher
You are right sorry about that. It won’t happen again.
no no worries, it was a suggestion, is not that you did something wrong, but it is possible that your PR doesn't get reviews and you get frustrated
/remove-sig node please re-add if this is node-related issue
I think it is a node related issue. The root cause in taint manager.
/sig node
/assign /triage accepted


Question: LocalModeInterfaceNamePrefix LocalMode = "InterfaceNamePrefix" ) <del> // IPVSSchedulerMethod is the algorithm for allocating TCP connections and // UDP datagrams to real servers. Scheduling algorithms are implemented as kernel modules. // Ten are shipped with the Linux Virtual Server. type IPVSSchedulerMethod string const ( // RoundRobin distributes jobs equally amongst the available real servers. RoundRobin IPVSSchedulerMethod = "rr" // WeightedRoundRobin assigns jobs to real servers proportionally to their real servers' weight. // Servers with higher weights receive new jobs first and get more jobs than servers with lower weights. // Servers with equal weights get an equal distribution of new jobs. WeightedRoundRobin IPVSSchedulerMethod = "wrr" // LeastConnection assigns more jobs to real servers with fewer active jobs. LeastConnection IPVSSchedulerMethod = "lc" // WeightedLeastConnection assigns more jobs to servers with fewer jobs and // relative to the real servers' weight(Ci/Wi). WeightedLeastConnection IPVSSchedulerMethod = "wlc" // LocalityBasedLeastConnection assigns jobs destined for the same IP address to the same server if // the server is not overloaded and available; otherwise assigns jobs to servers with fewer jobs, // and keep it for future assignment. LocalityBasedLeastConnection IPVSSchedulerMethod = "lblc" // LocalityBasedLeastConnectionWithReplication with Replication assigns jobs destined for the same IP address to the // least-connection node in the server set for the IP address. If all the node in the server set are overloaded, // it picks up a node with fewer jobs in the cluster and adds it to the sever set for the target. // If the server set has not been modified for the specified time, the most loaded node is removed from the server set, // in order to avoid high degree of replication. LocalityBasedLeastConnectionWithReplication IPVSSchedulerMethod = "lblcr" // SourceHashing assigns jobs to servers through looking up a statically assigned hash table // by their source IP addresses. SourceHashing IPVSSchedulerMethod = "sh" // DestinationHashing assigns jobs to servers through looking up a statically assigned hash table // by their destination IP addresses. DestinationHashing IPVSSchedulerMethod = "dh" // ShortestExpectedDelay assigns an incoming job to the server with the shortest expected delay. // The expected delay that the job will experience is (Ci + 1) / Ui if sent to the ith server, in which // Ci is the number of jobs on the ith server and Ui is the fixed service rate (weight) of the ith server. ShortestExpectedDelay IPVSSchedulerMethod = "sed" // NeverQueue assigns an incoming job to an idle server if there is, instead of waiting for a fast one; // if all the servers are busy, it adopts the ShortestExpectedDelay policy to assign the job. NeverQueue IPVSSchedulerMethod = "nq" ) </del> func (m *ProxyMode) Set(s string) error { *m = ProxyMode(s) return nil

The ipvs mode limit the scheduler algorithm to : now in new kernel have this new algorithm : the are specialy interesting... that be in kernel 4.18...
/sig network /area ipvs
/assign
This seems non-controversial on its face, but a) how do we know whether the kernel supports it or not; and b) if every node makes independent decisions, doesn't this basically become random? /assign uablrek
a) It should be the user's responsibility to ensure that the configured ipvs sheduler is available on all nodes This sounds rude, but please read for some rationale (and ). Exception may be the which is the default, but an installation with but without the sheduler is extremely unlikely. If that relaxing of the sheduler pre-test is acceptable, I would say that any current and future ipvs sheduler shall be allowed. Just treat them as a string. But it should be documented in the release-notes and some place else, but I don't know where.
b) For all shedulers that make decisions based on local state, yes. (that includes btw) Well, not really random ... For instance if is used and a new endpoint is addred, then all ipvs instances will get an "empty" target and will select it for new connections and may overload the newcomer, or it will become "statistically correct" over time. But IMHO users should be allowed to try. Some sheduler may be very useful in their environment. That will (already has) of course spawn some issues when it doesnt work as expected (for the reason in b) for instance), but allowing all shedulers will not change that.
So, my proposal is: Drop the scheduler module check Treat the configured scheduler as a string in the ipvs proxier Document
The module check can be replaced with a function check, e.g. insert/delete an ipvs dummy rule.
people is trying to reduce privileges on containers these days, does it requires more privileges than today or is the same?
The same. Kube-proxy configures ipvs already. This would only add an extra probing insert/delete on start. Even on first start (if ipvs is empty) with some care.
Could even be less privileges if "modprobe" requires any privileges (now or in the future)
What is your use-case?

I have wanted to check the sceduler in ipvs, but havn't got time yet. The load-balancer is stateless, but is not (well, at least not normally). A problem Maglev tries to solve is if the number of Application Servers (AS) (endpoints in K8s) changes, the updated hash buckets must be as few as possible. But if the load-balancer is stateful and stores the connections (like ipvs or the Linux conntrack used in proxy-mode=iptables), no existing connections are affected, no matter what hash algorithm is used for new connections. So, for both proxy-mode ipvs and iptables existing connection are not affected by adding or removing endpoints. But... if external traffic is routed to different nodes, e.g. a GW rehashes ECMP targets after a cluster scaling, then traffic is affected. This is easily tested: Setup a service of type LoadBalancer Route external traffic to the loadBalancerIP to node A Setup a number of TCP connections Re-route external traffic to the loadBalancerIP to node B All connections are lost (I have tested). But if the load-balancer would have been a stateless Maglev no connections would be lost. Google Maglev stores connections but only as a cache and getting the best of stateful, existing connections never lost on AS scaling, and stateless, the load-balancing tier can be scaled with minimum impact. This can be the case in ipvs , but is it?
Lars: proposal SGTM. IPVS impl is pretty much your call to make :) On Thu, Dec 22, 2022, 1:37 AM Lars Ekman
I have started working on a PR to remove the module checks. You are the only active reviewer and I will try my best to divide it into small reviewable commits as you wrote in :smile:
PR to remove the module checks I also intend to remove the constraint on ipvs schedulers, which will fix this issue. If you can create a VS with your configured scheduler you can use it, whatever that scheduler may be.
Looks like it is. Please read I have tested "mh" in kube-proxy (with a patch) and first problem is that is hashes on address only by default, so a traffic generator using the same source all the time will end up on the same endpoint.

This can be configured but will require special treatment in kube-proxy.
Should we close this now that is in?
/triage accepted
No, just removes the module ckeck. The validator still restricts the usable schedulers
to add afinity without using states. is the finality of maglev to avoid perf on managing state... and is more robust on target lost than other consistant hashing method... is normal is what you whant when you use maglev... to have afinity on ip src... but without need of storing states... is afinity by consistant hashing... cilium use this algorithm too....
the google research paper on the algorythm:
If you can build and use an own please try PR I have tested and "mh" can be configured, and with many src-addresses load-balancing works. However if external ingress traffic is re-routed from one K8s node to another, all connections are lost, same as "rr".
with ipvs the node who receive the traffic is considered a director, if you LB to other node... this node must have default route to the director because of nat masquerading of ipvs... when you use MH/RR with Direct Server Return avoid that problem (no more nat masquerading)... the node respond to the client directly (no need to return back to the first node who receive the traffic). but you need to tunnel the traffic from node who receive the request to the node you forward the traffic (cloud flare use fou protocol for that)... i dont know if kubeproxy already support that sort of configuration... cilium for LB DSR + Maglev use tcp option field to store the Virtual IP that receive the traffic in the packet and nat the destination with the pod ip before sending to the node who host the node (that work in full routing mode), and the node respond to the client directly in replacing thaire ip with the Virtual IP from tcp option field.... in that way no state needed... no tunneling... no complex network configuration.... some has been done on kube-router project: and i think you must use in conjonction with MH/RR to use Direct Server Return if i understand whell... and good exemple of the use case : that use ipvs (use ipip tunnel) to do that....
Regarding "mh" algorithm, I submitted PR a few weeks ago which adds the new load distribution method to kube-proxy.

As this commit supports 'mh" with hashing on srcIP and srcPort by default, connections from the same srcIP with different srcPorts will be distributed equally to all EndPoints. If users want to have an affinity on srcIP, they can use "mh" with "Session Affinity" option enabled. The PR is targeted for the main use-cases below: Connection failover from a worker node to another. Scale-in/out worker nodes Rerouting from GWs or external LoadBalancer changes receiving worker nodes of existing traffic flows. I would need to rebase the commit after your fix for this issue PR . PR is now waiting triage state, please verify if it is reasonable to add this new algorithm to kube-proxy.
No, and it probably never will. The proper (only) way to get this into is to write a (KEP). To :
I would also like to see a demo or e2e test working with this setup, it doesn't seem obvious from the comments that this will work out of the box ... The project Kubernetes, as Lars correctly mentions , use KEPs to discuss and document feature, I agree with him and I think this will require to write a KEP
What I meant was your use-case in K8s. Having worked most of 2021 with the Maglev hash algorithm when developing the I know what it can do, but I also know K8s and I think I can say right now; To allow the "mh" in kube-proxy=ipvs will not give you what you hope for.
And a side-remark on DSR. Cilium can do this because they modify packets with eBPF. The traditional way of doing DSR is to actually assign the Virtual IP address (VIP) to an interface in the server (the POD in the K8s case). This must have support in the CNI-plugin, which I guess is the case for as it handles load-balancing and is a CNI-plugin. To add generic support for DSR in K8s is not simple.
I'm new to this community. Thanks for your suggestion about the process of adding a new feature to kube-proxy. I'll prepare a KEP for the "mh" load distribution.
But in any case I would like PR to be merged. For sanity reasons if not others.

The for scheduler=mh can/should be in a separate issue+PR. This will allow users to test "mh" in kube-proxy, because; Nobody will take my word for it :smile:
I don't want to sound discouraging but IMO you should not spend too much time on a KEP before presenting it to get some early feedback. I can't see how "mh" can provide the statelessness unless DSR is used, and that may be very hard to get into the . So, I suggest you describe a use-case in the KEP, and what it takes to support it, and present it on a bi-weekly sig/network meeting. And don't worry, people are nice there :smile:
Perhaps DSR is not needed to use with "mh" since I cannot really see any problem with nat masquerading of ipvs if subsequence packets are sent to the second node. With net/ipv4/vs/sloppy_* enabled, ipvs on the second node accepts the packets as an existing connection and then keeps forwarding them to the original endpoint thanks to the mh algorithm. IPVS of course performs snat and "remember" the client IP of the packets as if they are sent to the first node. When receiving replies from the endpoint, ipvs on the second node will be able to forward the replies to the client with the cached client IP in its NAT table. Please correct me if I'm missing something in this regard.
So even if the kube-proxy forwards packets to the same POD (assuming the same mh configuration in node A and B) the receiving POD will reject with a TCP-RST packet.
Since you have written PR you should be able to test this with you own-built kube-proxy. For me all connections are teminated when re-directing ingress traffic from one K8s node to another.
Thanks for your clarification. Now I could see this problem is on the pod side, not ipvs. It seems that having DSR does not help to solve the problem. Though, I suppose this issue will not happen with the following scenarios: UDP traffic Traffic is sent to Cluster-IP instead of NodeIP
With DSR the src=VIP all the way, so it should be OK, even if masq (DNAT) is used.

UDP same thing, unless your UDP server has no session and doesn't care about addresses, just the server port. This is however not the usual case. Traffic sent to the Cluster-IP will be have src=pod, yes, but it will always be hadled by the local kube-proxy, so the use-case to move traffic to another node doesn't exist for clusterIP.
Me neither :) Agree - you will have the best experience if do this iteratively. Start with an email to the SIG list. Then a doc discussing the peoblem and high-level solution details. THEN a KEP makes sense.
cilium assign vip also but more higher in the kernel stack with ebpf... in that way is more simple to implemente than in ipvs... juste simple packet manipulation (plus the fact that is working only in full routed mode)... ipvs can do that, helped with some config on each node (configuring tunnel for dsr)... for mh usage without dsr we need to add default route on other node to the actual dedicated kubeproxy as l4 director (one node can be designated as director and move to another node when the first one die). is generaly more for loadbalancer/external ip usage....
Is this an acceptable way? merged. This will allow "mh" to be configured as a ipvs scheduler, but only with L3 hashing (no ports). a new issue like "Ipvs scheduler=mh only hashes on addresses, not ports", and implement it in a new (small) PR with FlagSourceHash as have already done in this issue as "done" since "mh" can be configured and as it's implemented in another way. the work towards a KEP to make "mh" work as you want it to work. Including a K8s use-case, DSR (if needed in some form), setting of sysctl's, etc. I leave item 4. to you and . Item 1-3 will enable "mh" but because of NAT, connections are still lost when traffic is re-directed from one node to another (this can be tested without too much effort. I have done it).
Agree with the approach. fixes the raised issue that it enables all distribution methods supported by ipvs in kube-proxy (including "mh" one).

A separate issue for enabling source port hash when using 'mh' sounds reasonable. This is not only applicable for "mh", but also for the "sh" method which kube-proxy is currently supporting as stated in k8s documents. Agree. Sure. I'll need to verify the issue further and check if any easy way possible to avoid the NAT issue on different nodes. IMHO, replacing VIP (virtual IP) with NodeIP seems to be the first option to overcome the issue, but we need to verify its feasibility and impact in k8s.
To allow the "mh" in kube-proxy=ipvs will not give you what you hope for. Nobody will take my word for it :smile: Ok, ok, please let me rephrase: To just allow "mh" in the current ipvs proxier will not give you what you hope for. So, is not enough.
/kind feature


Question: // CauseTypeFieldValueNotSupported is used to report valid (as per formatting rules) // values that can not be handled (e.g. an enumerated string). CauseTypeFieldValueNotSupported CauseType = "FieldValueNotSupported" <ins> // CauseTypeForbidden is used to report valid (as per formatting rules) // values which would be accepted under some conditions, but which are not // permitted by the current conditions (such as security policy).  See // Forbidden(). CauseTypeForbidden CauseType = "FieldValueForbidden" // CauseTypeTooLong is used to report that the given value is too long. // This is similar to ErrorTypeInvalid, but the error will not include the // too-long value.  See TooLong(). CauseTypeTooLong CauseType = "FieldValueTooLong" // CauseTypeTooMany is used to report "too many". This is used to // report that a given list has too many items. This is similar to FieldValueTooLong, // but the error indicates quantity instead of length. CauseTypeTooMany CauseType = "FieldValueTooMany" // CauseTypeInternal is used to report other errors that are not related // to user input.  See InternalError(). CauseTypeInternal CauseType = "InternalError" // CauseTypeTypeInvalid is for the value did not match the schema type for that field CauseTypeTypeInvalid CauseType = "FieldValueTypeInvalid" </ins> // CauseTypeUnexpectedServerResponse is used to report when the server responded to the client // without the expected return type. The presence of this cause indicates the error may be // due to an intervening proxy or the server software malfunctioning.

from: from: CauseType and field.ErrorType should be exactly mapped according to the comment of ErrorType, but currently this is not the case, metav1.CauseType is missing several values none No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted


Question: if spec.Template.Spec.RestartPolicy != api.RestartPolicyAlways { allErrs = append(allErrs, field.NotSupported(fldPath.Child("template", "spec", "restartPolicy"), spec.Template.Spec.RestartPolicy, []string{string(api.RestartPolicyAlways)})) } <ins> if spec.Template.Spec.ActiveDeadlineSeconds != nil { allErrs = append(allErrs, field.Invalid(fldPath.Child("template", "spec", "activeDeadlineSeconds"), spec.Template.Spec.ActiveDeadlineSeconds, "must not be specified")) } </ins> allErrs = append(allErrs, apivalidation.ValidateNonnegativeField(int64(spec.MinReadySeconds), fldPath.Child("minReadySeconds"))...) allErrs = append(allErrs, apivalidation.ValidateNonnegativeField(int64(spec.TemplateGeneration), fldPath.Child("templateGeneration"))...)

setting activeDeadlineSeconds in deployment pod template, and seen new pods are continuously created after the older one dies. Is this excepted? if not we should forbidden this.
Yes, it is expected. Once a pod dies, the replica set controller will spawn a new one. activeDeadlineSeconds doesn't make any sense when used under a controller.
we may want to make it a validation error, but I don't feel strong.
Yes, we should not allow activeDeadlineSeconds for restartPolicy=Always pods generally.
What if I want a container to be restarted no matter what status it exits with, but to give up trying if it cannot even start up in seconds? So like this: Start a pod if it keeps constantly failing for seconds: stop it else (if it managed to start and not fail in those first seconds from creating): restart it everytime it fails or successfully finishes
then you want your own controller, RCs are designed for a simple use case: long-running apps. Have a look in , though keep in mind they are under a major redesign:
Sorta late to the party, but this behavior stops us from being able to use a Deployment to mitigate a production memory leak. Our application can handle a , but a leaves items in a pending state for our customers. Killing pods on a regular basis through kubernetes would allow us to induce known behavior of a graceful termination versus deploying new code to handle a timer before terminating. While I understand the intention of this feature, I feel that as a mitigation tool while my teams track down the memory leak, this would let my oncall rotation sleep easy.


Question: EOF fi <ins> install-salt </ins> <del> # Install Salt # # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 curl -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -X </del> <ins> service salt-minion start </ins>

I saw this error in on the master after a failed cluster start: I'm looking into this, but I want to create a tracking issue as I think some other people may be hitting this.
It's trying to bring up and failing: The weird thing is that neither that file nor the file have been modified that recently:
Yes, to me it looks like salt just isn't finding the plugin. The last change to _states did add , but I've seen this succeed recently as well (http://kubernetes-). I'm going to try this a few times, to try to figure out whether it is failing every time or not. My guess is that it is failing semi-randomly, so it isn't related to the changes since last-fail.
Aha... this is logged at debug level from I'm guessing it's a salt version mismatch... investigating.
I guess AWS needs to lock salt versions like GCE does:
Confirmed that this is because of upstream salt breakage: - deprecated (?) Not sure what we should do here. I can lock to the same version as GCE, but if we find a bug in that version of salt then we can't update without breaking this script.
We don't write bugs. ;)
Can you just put this in an AWS-specific block? It seems like that should work.
(And by "this" I mean "future Salt variable gunk")
I think I'm going to try to lock AWS to the same version as GCE, and then think more about a long-term plan once AWS is working again!
from the looks of things I think vagrant will have the same problem (?) (Note that I am in the process of developing / testing the branch above, so I don't think you can just copy-and-paste the fix - yet!)
As a short term fix, using 2015.05 (ubuntu trusty salt ppa), was to change from: import as ipaddr to: import as ipaddr this actually seems to be working. might be deprecated. but not in 2015.05. why is there a containerbridge custom state at all btw? why not use network.managed and
- I'm not sure we can just change the import, because I think we have to be compatible with GCE's version of salt also.

I'm not sure of why we did it this way vs other ways of doing things in salt
Does this mean that the official k8s distribution is broken on AWS, BTW? When is the next point release (and where are these discussed)?
We cherry-pick into point releases as necessary, and it's been pretty informal and usually initiated by us (Google). I'd actually be happy to have a discussion about how to initiate it externally (we're going to need a policy for requesting such things). cc
FWIW, policy aside, I'm happy to cut a point release for this once you think is tested well enough.
That would be great, thanks. I'll be testing it pretty hard today, I think. It has occurred to me that other clouds may also be broken. By grepping, it looks like vagrant and vsphere use I am utterly unfamiliar with the vsphere implementation, and I think vagrant sidesteps the issue because on Redhat is skips the call to It might be good to get confirmation that vagrant hasn't fallen in to a different trap though! Let me test it out today (unless people start filing lots of issues, in which case I think we roll it out more urgently...)
I will test out the Vagrant setup today, but I think we should be good because we do not use the same networking setup.
File a bug for a 0.17.1 to get this cherry-picked and assign it to me, as a reminder?
Do you know the status of 17.1? I'm seeing the same problems trying to AWS working, even tried an older release (e.g. v15), and have the same problem. My script looks 32 times on "[salt-master not working yet]" then gives up.
We're cutting one today.
I'm still seeing the same issue as before with the new release, my master starts then times out waiting for salt-master to start. Do you know some spots I can look at to help troubleshoot? Do you want a new issue created to track against?
Please file a new issue and tag thanks. I'm sorry it didn't fix things for you! On May 22, 2015 4:49 AM, "Steve Sloka" wrote:


Question: return nil } <del> func deleteNamespaceFunc(cs clientset.Interface, ns string, timeout time.Duration) func() { return func() { err := cs.CoreV1().Namespaces().Delete(context.TODO(), ns, metav1.DeleteOptions{}) if err != nil && !apierrors.IsNotFound(err) { framework.Logf("error deleting namespace %s: %v", ns, err) } err = framework.WaitForNamespacesDeleted(cs, []string{ns}, timeout) if err != nil { framework.Logf("error deleting namespace %s: %v", ns, err) } } } </del> func tryFunc(f func()) error { var err error if f == nil {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-ubuntu-gce-containerd Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Pre-provisioned PV (default fs)] subPath should support file as subpath [LinuxOnly] Testgrid link: Reason for failure: Will look at it shortly, why it is flaking. Anything else we need to know: links to appreciatedlinks to specific failures in spyglass appreciated
/sig storage
The namespace terminating during test setup is super strange but seems like a test-only issue, not an issue with the functionality being tested. I'd recommend looking at the test, API server, and controller logs to figure out why the test namespace () was deleted and who deleted it
does this help ?
feels like a test only problem like said. We don't even have gluster in our CI/prow to test against... cc Michelle, Xing, Can you please confirm that this is not a blocker for 1.19.0?
Knowing the timeframe for when the test started and when that delete happened would be interesting to know. Possibilities I can see: something in the test timed out, and the cleanup happened in parallel with something in the test attempting to run multiple tests are reusing the same test context and namespace name at the same time (I can never keep straight how ginkgo parallel tests work)
This does not look like a blocker for 1.19.0.
Thanks /milestone v1.20
looking at the logs it seems to be the latter "same test context and namespace name at the same time", but we can deal with this for 1.19.1
cc This looks like another instance of "the same test namespace is being reused". I believe had a mitigation for that to try to make the namespaces more unique.
(btw, the test brings up a containerized gluster server so we are exercising gluster functionality)
Yeah, looking at the build log found that it is the same issue as The issue is still around the double deletion of namespace, i.e. we delete the test namespace during the test execution, but later on the call we delete the namespace again to ensure it truly get cleaned up.

But unfortunately, this namespace has been reused by another test case between this gap.
Some potential fix I can think of: deletion of the primary namespace twice is inevitable, maybe we should consider generate primary namespace based on a hashing of the test cases name? the 4 digits random namespace to 6, this should not solve the problem in nature but it will extremely decrease the chance it happens. using an in-memory namespace map to keep record of already generated namespaces for the whole test run and refuse to generate the same namespace again in the whole lifecycle of a prow job.
do you know if this gluster test is using your mitigation in ?
hash of the testcase name + random suffix (because tests can be run multiple times) seems like a good way to mitigate the chances of tests reusing the same namespace. I'm unsure if we know the test case name at the time of test namespace creation though. I think increasing the length of the random suffix is a quick and easy mitigation believe ginkgo to run in parallel, so this may not work
Yes it is. But my fix in can only reduce the chance of collision and probably improve readability of logs. But not to eliminate this issue in nature.
AFAICT does not affect intree code at all, because it only fixes namespace of driver. If we are seeing collision of primary namespace for intree code then it is a different problem.
on only deals with the collision between test namespace and driver namespace. And that fix is taking affect. failure is because a test namespace collides with another test namespace. And this is not related to in-tree code at all. Any test could potentially become flaky because their namespace can be deleted during test execution. It's just happens to be this intree test this time. The root cause for this problem is: We cleaned up the test namespace twice which gives it a small gap that another testcase can generate the same namespace and be affected by the second namespace deletion from the original testcase.
/assign I would like to give it a shot.
I've opened to address this issue.
/unassign /assign


Question: nodeInfo: schedulercache.NewNodeInfo( newResourcePod(schedulercache.Resource{MilliCPU: 2, Memory: 2, StorageOverlay: 5})), fits: false, <del> test: "request exceeds allocatable", </del> <ins> test: "request exceeds allocatable overlay storage resource", reasons: []algorithm.PredicateFailureReason{ NewInsufficientResourceError(v1.ResourceStorageScratch, 18, 5, 20), }, }, { pod: newResourcePod(schedulercache.Resource{StorageOverlay: 18}), nodeInfo: schedulercache.NewNodeInfo( newResourcePod(schedulercache.Resource{MilliCPU: 2, Memory: 2, StorageOverlay: 5})), fits: false, test: "request exceeds allocatable overlay storage resource", </ins> reasons: []algorithm.PredicateFailureReason{ NewInsufficientResourceError(v1.ResourceStorageScratch, 18, 5, 20), },

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): local isolation Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Bug Kubernetes version (use ): kubernetes HEAD Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: For local storage isolation, overlay resource predicates for pod with only overlay or scratch storage request doesn't work. What you expected to happen: If a pod requests only overlay or scratch storage, then local storage isolation feature should limit the pods. How to reproduce it (as minimally and precisely as possible): Run local cluster from master branch, and create following pods (against a machine with less overlay fs than 1024Gi). Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage
CC
Fix is ready:


Question: `Username          jane.doe`, `UID               uniq-id`, `Groups            [students teachers]`, <del> `Extra: subjects   [math sports]`, </del> `Extra: skills     [reading learning]`, <ins> `Extra: subjects   [math sports]`, </ins> ``, }, },

Unit tests failed with , see TestWhoAmIRun/successtest None No response No response_ /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I think it was caused by
When we loop the golang Map, it's unordered.
I can work on this one.
Thanks.
/priority critical-urgent this is impacting CI,


Question: storage, _, server := NewTestREST(t, nil, tc.enableDualStack) defer server.Terminate(t) <ins> if tc.isIPv6Primary { if storage.secondaryServiceIPs != nil { storage.serviceIPs, storage.secondaryServiceIPs = storage.secondaryServiceIPs, storage.serviceIPs } else { r, err := ipallocator.NewCIDRRange(makeIPNet6(t)) if err != nil { t.Fatalf("cannot create CIDR Range(primary) %v", err) } if tc.enableDualStack { storage.secondaryServiceIPs = storage.serviceIPs } storage.serviceIPs = r } } </ins> if tc.enableDualStack && storage.secondaryServiceIPs == nil { t.Errorf("storage must allocate secondary ServiceIPs allocator for dual stack") return

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: framework.RunKubectlOrDie(ns, "run", podName, "--image="+httpdImage, "--labels=run="+podName) ginkgo.By("replace the image in the pod with server-side dry-run") <del> podJSON := framework.RunKubectlOrDie(ns, "get", "pod", podName, "-o", "json") podJSON = strings.Replace(podJSON, httpdImage, busyboxImage, 1) if !strings.Contains(podJSON, busyboxImage) { framework.Failf("Failed replacing image from %s to %s in:n%sn", httpdImage, busyboxImage, podJSON) } framework.RunKubectlOrDieInput(ns, podJSON, "replace", "-f", "-", "--dry-run=server") </del> <ins> specImage := fmt.Sprintf(`{"spec":{"containers":[{"name": "%s","image": "%s"}]}}`, podName, busyboxImage) framework.RunKubectlOrDie(ns, "patch", "pod", podName, "-p", specImage, "--dry-run=server") </ins> ginkgo.By("verifying the pod " + podName + " has the right image " + httpdImage) pod, err := c.CoreV1().Pods(ns).Get(context.TODO(), podName, metav1.GetOptions{})

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Also flaking on: Which test(s) are failing: Since when has it been failing: ~Since Oct 21? Testgrid link: CAPG conformance: CAPA conformance: gce cos master default: Reason for failure: Example spyglass link: Anything else we need to know: Triage shows that this is happening across several jobs (not just the ones I listed above) and the spike started happening earlier this week. /sig cli /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fixed by
This is affecting ppc64le jobs as well.
/remove-kind failing-test /kind flake This is not a continuously failing test, it is instead flaking
/priority critical-urgent the top flake for per /area conformance this is a conformance test, it definitely shouldn't be flaking, and should be demoted if it can't be fixed
/reopen For CI signal to observe before marking as resolved
Reopened this issue. <detailsIn response to : /reopen For CI signal to observe before marking as resolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Last failure 7 days ago 2020-11-03 05:40:32 +0000 UTC
Please review to see if we can close this now.
Let's close this - also looks like it is no longer flaking as well


Question: "k8s.io/kubernetes/pkg/registry/generic/registry" "k8s.io/kubernetes/pkg/registry/namespace" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/storage" </ins> ) // rest implements a RESTStorage for namespaces against etcd

Honestly I'm not sure if this is a bug or a feature, but didn't expect it. Filing this to track.
I think it's a bug. Nothing in the code base depends on it that I can see (edit: namespace deletion does, but only via an in-server call which I'd like to see fixed), and we don't let other delete-related fields mutate during update (deleteGracePeriodSeconds)


Question: command := exec.Command("umount", target) output, err := command.CombinedOutput() if err != nil { <del> glog.Errorf("Unmount failed: %vnUnmounting arguments: %snOutput: %sn", err, target, string(output)) return err </del> <ins> return fmt.Errorf("Unmount failed: %vnUnmounting arguments: %snOutput: %sn", err, target, string(output)) </ins> } return nil }

We have to spend times on figuring out what are the mount error code means from issues and . In , the error message is and is great for diagnosing mount problem. The console message better be returned rather than just error code.


Question: indexer.Add(&v1.Node{ObjectMeta: metav1.ObjectMeta{Name: nodeName}}) } <ins> esInformer := informerFactory.Discovery().V1().EndpointSlices() esIndexer := esInformer.Informer().GetIndexer() // These reactors are required to mock functionality that would be covered // automatically if we weren't using the fake client. client.PrependReactor("create", "endpointslices", k8stesting.ReactionFunc(func(action k8stesting.Action) (bool, runtime.Object, error) { endpointSlice := action.(k8stesting.CreateAction).GetObject().(*discovery.EndpointSlice) if endpointSlice.ObjectMeta.GenerateName != "" { endpointSlice.ObjectMeta.Name = fmt.Sprintf("%s-%s", endpointSlice.ObjectMeta.GenerateName, rand.String(8)) endpointSlice.ObjectMeta.GenerateName = "" } endpointSlice.Generation = 1 esIndexer.Add(endpointSlice) return false, endpointSlice, nil })) client.PrependReactor("update", "endpointslices", k8stesting.ReactionFunc(func(action k8stesting.Action) (bool, runtime.Object, error) { endpointSlice := action.(k8stesting.CreateAction).GetObject().(*discovery.EndpointSlice) endpointSlice.Generation++ esIndexer.Update(endpointSlice) return false, endpointSlice, nil })) </ins> esController := NewController( informerFactory.Core().V1().Pods(), informerFactory.Core().V1().Services(), nodeInformer, <del> informerFactory.Discovery().V1().EndpointSlices(), </del> <ins> esInformer, </ins> int32(100), client, batchPeriod)

The controllers can create a second EndpointSlice after an immediate sync if the first EndpointSlice they created is not in the cache yet. Here's what can happen: gets event that triggers sync for Service syncs and creates EndpointSlice gets event that triggers sync for same Service syncs and creates EndpointSlice (the slice created in 2 has not made it to the cache yet) A single EndpointSlice to be created. Run integration tests hundreds of times that involve 2 consecutive events that will trigger a sync. Example: /assign /sig network
/triage accepted
/cc
When this bug is fixed, we'll need to rollback
just amend it in the PR that fix the bug, so we merge all together


Question: t.Errorf("diff returned %v", diff) } } <ins> func TestCompileRegex(t *testing.T) { uncompiledRegexes := []string{"endsWithMe$", "^startingWithMe"} regexes, err := CompileRegexps(uncompiledRegexes) if err != nil { t.Errorf("Failed to compile legal regexes: '%v': %v", uncompiledRegexes, err) } if len(regexes) != len(uncompiledRegexes) { t.Errorf("Wrong number of regexes returned: '%v': %v", uncompiledRegexes, regexes) } if !regexes[0].MatchString("Something that endsWithMe") { t.Errorf("Wrong regex returned: '%v': %v", uncompiledRegexes[0], regexes[0]) } if regexes[0].MatchString("Something that doesn't endsWithMe.") { t.Errorf("Wrong regex returned: '%v': %v", uncompiledRegexes[0], regexes[0]) } if !regexes[1].MatchString("startingWithMe is very important") { t.Errorf("Wrong regex returned: '%v': %v", uncompiledRegexes[1], regexes[1]) } if regexes[1].MatchString("not startingWithMe should fail") { t.Errorf("Wrong regex returned: '%v': %v", uncompiledRegexes[1], regexes[1]) } } </ins>

In case it isn't clear why this happened, it's because gcloud ships with an older version of kubectl (currently version 0.7.0). We keep it as up to date as we can, but it will always lag a release or so behind head.
Fixed by the revert.


Question: // do not, the taint will be removed, and this will not be retried return nil, err } <ins> } else { providerID = node.Spec.ProviderID </ins> } <del> nodeAddresses, err := getNodeAddressesByProviderIDOrName(ctx, instances, node) </del> <ins> nodeAddresses, err := getNodeAddressesByProviderIDOrName(ctx, instances, providerID, node.Name) </ins> if err != nil { return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: introduces a regression that func is invoked on a Node obj without , hence it can fail fetching the node addresses when the is not set. What you expected to happen: I would expect cloud-controller-manager to be able to initialize new Node. How to reproduce it (as minimally and precisely as possible): the gcp legacy-cloud-controller v1.18.1 () that the node-controller cannot initialize a new Node. Logs of the cloud-controller-manager repeated multiple times: Ensure that the Node is with the taint. Ensure that the of the Node is not set. The cloud-controller-manager is not running on the host machine and that's why it cannot call the gce matadata. But it was never a hard requirement for cloud-controller-manager to run on the host machine. Anything else we need to know?: The legacy cloud-controller-manager v1.17.x is working perfectly fine. Environment: Kubernetes version (use ):v1.18.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /kind regression
/cc
can we open a new issue re:


Question: // For each node, if the node is running the daemon pod but isn't supposed to, kill the daemon // pod. If the node is supposed to run the daemon pod, but isn't, create the daemon pod on the node. var nodesNeedingDaemonPods, podsToDelete []string <del> var failedPodsObserved int </del> for _, node := range nodeList { <del> nodesNeedingDaemonPodsOnNode, podsToDeleteOnNode, failedPodsObservedOnNode, err := dsc.podsShouldBeOnNode( </del> <ins> nodesNeedingDaemonPodsOnNode, podsToDeleteOnNode, err := dsc.podsShouldBeOnNode( </ins> node, nodeToDaemonPods, ds) if err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I have a daemonsets with 250 replicas and maxUnavailable is 20%. When I try to update its image, the ds will scale down 50 old replicas and wait for SatisfiedExpectations, after that it will scale up 50 new replicas. But if a pod delete failed (node is notready, and pod stuck at terminating status), ds controller will never reach the expectation and just stuck here. What you expected to happen: When a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-8+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-25T13:04:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig apps /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
DaemonSet controller waits for old pods to terminate before creating new ones for best-effort at-most-one pod per node. This is a deliberate design decision.

In most cases, DaemonSets won't tolerate two pods scheduled on the same node at the same time; for example, it's common to communicate with DaemonSet pods via . I'm going to close this ticket as this works as designed. Please feel free to reopen the ticket if you think there are still issues to be resolved.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
why does the DaemonSet controller wait for all pods 50 to be done, before moving to the NEXT 50 ? I completely understand that we want only at most 1 pod per node. But if a pod deletion fails on the node, move on to the next pod as long as the total unavailable pods are less than maxUnavailable. This will allow us to make progress in the face of bad nodes or failed pod deletions due to failing tear down of bad volumes or other reasons.
/assign
thoughts on this one ?
Any progress on this issue? We also encountered this problem. There are thousands of nodes in our prod cluster and nodes often go down, which cause daemonset rolling update stucked, this problem really bother us.
PTAL
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Given that the Job already has the SuccessCriteriaMet condition, the termination condition already had confirmed in another cycle. // So, the job-controller evaluates the podFailurePolicy only when the Job doesn't have the SuccessCriteriaMet condition. if jobCtx.finishedCondition == nil && feature.DefaultFeatureGate.Enabled(features.JobPodFailurePolicy) { <del> if failureTargetCondition := findConditionByType(job.Status.Conditions, batch.JobFailureTarget); failureTargetCondition != nil { </del> <ins> failureTargetCondition := findConditionByType(job.Status.Conditions, batch.JobFailureTarget) if failureTargetCondition != nil && failureTargetCondition.Status == v1.ConditionTrue { </ins> jobCtx.finishedCondition = newFailedConditionForFailureTarget(failureTargetCondition, jm.clock.Now()) } else if failJobMessage := getFailJobMessage(&job, pods); failJobMessage != nil { // Prepare the interim FailureTarget condition to record the failure message before the finalizers (allowing removal of the pods) are removed.

When the FailureTarget=False condition is manually, then the Job terminates. The condition should be ignored by the Job controller. the like this: the following script to repro: Issue: the Job gets terminated with the status like this: No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign Thanks!
/sig apps


Question: Status: v1.PodStatus{ NominatedNodeName: "node1", }, <del> }), framework.NewPodInfo(&v1.Pod{ </del> <ins> }) medPriorityPodInfo = framework.NewPodInfo(&v1.Pod{ </ins> ObjectMeta: metav1.ObjectMeta{ Name:      "mpp", Namespace: "ns2",

The scheduler cache entered a corrupted state where a Pod, according to the cache dump, was nominated to node A, while the status said it was nominated to node B. This lead to unschedulable low priority pods due to the phantom pod in the nominator cache. The cache to match the Pod status Setup: two nodes with all the resources used. A high priority pod comes in that is able to preempt at least one pod. This might cause these sequence of events X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there. the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Nothing clears this nomination afterwards Usually step 2 happens after 3, so everything works as expected. But in a busy cluster, 3 might be delayed. No response <detailsGKE </details<details</details<details</details<details</details<details</details
/sig scheduling cc
This Pod actually got scheduled: The cache dump was obtained much later at : I think the problem is that we got an update for the Pod (probably the nominated node itself), scheduled it again and assigned a second nominated node for it. The second scheduling attempt failed, but we didn't clear the cache.
/triage accepted
/priority important-soon
Sounds like is related?
It could be the same, or it could be another bug. Note that this is a second scheduling attempt for a Pod that was already scheduled.
My theory of events: X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there.

the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Not sure why this pod update fails, but nevertheless, we have a corrupted cache at this point.
thanks for raising up this and the fix. Could you update the "How can we reproduce it (as minimally and precisely as possible)?" section a bit? Per our offline conversation, the issue will leave a "phantom" pod holding a number of resources in the internal nominator cache. And due to that unnecessary resource reservation, low-priority pods may not get scheduled while they should have been. This might be the key trigger of this issue. In practice, it can be caused by a (way-too-quick) movePodBackToActiveQ call (due to relevant/irrelevant events, or backoff's completion) after step 1 and before step 3.
Updated the description. I will update the PRs on Monday.


Question: } else { tc.Unlock() } <del> // Wait for HPA to be processed. <-tc.processed </del> tc.verifyResults(t) } func TestLegacyScaleUp(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Resync period for HorizontalPodAutoscaler controller is set to 15 seconds in order to update HPAs every 15 seconds. Sometimes updates are skipped and period between two updates is 30 seconds. What you expected to happen: HPA should be updated every 15 seconds. How to reproduce it (as minimally and precisely as possible): Add debugs to HPA controller showing how often HPAs are processed. Create HPA. Anything else we need to know?: HPA controller uses rate RateLimitingQueue: New requests are always with 15 seconds delay: If request for the given HPA is in the queue then newly request is skipped. Requests are and removed from the queue at the same moment so there is a race condition. See following scenario: 00:00:00 - Resync adds request to queue to be executed at 00:00:15 00:00:15 - Resync adds next request to queue to be executed at 00:00:30 but it is skipped because previous request is still in the queue 00:00:15 - Request at 00:00:00 is executed 00:00:30 - Resync adds next request to queue to be executed at 00:00:45 00:00:45 - Request at 00:00:30 is executed In above scenario HPA wasn't updated between 00:00:15 and 00:00:45. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration: GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig autoscaling
/assign krzysztof-jastrzebski


Question: }, }, }, <ins> { Pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "node_info_cache_test", Name:      "test-3", UID:       types.UID("test-3"), }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceCPU: resource.MustParse("200m"), }, }, Ports: []v1.ContainerPort{ { HostIP:   "127.0.0.1", HostPort: 8080, Protocol: "TCP", }, }, }, }, InitContainers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("500m"), v1.ResourceMemory: resource.MustParse("200Mi"), }, }, }, }, NodeName: nodeName, Overhead: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("500m"), v1.ResourceMemory: resource.MustParse("500"), }, }, }, }, </ins> }, }

The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!


Question: } endpoints := make([]string, len(pods)) for ix, pod := range pods { <del> // TODO: Use port names in the service object, don't just use port #0 endpoints[ix] = net.JoinHostPort( pod.CurrentState.Host, strconv.Itoa(pod.DesiredState.Manifest.Containers[0].Ports[0].HostPort), ) </del> <ins> port, err := findPort(&pod.DesiredState.Manifest, service.ContainerPort) if err != nil { glog.Errorf("Failed to find port for service: %v, %v", service, err) continue } endpoints[ix] = net.JoinHostPort(pod.CurrentState.PodIP, strconv.Itoa(port)) </ins> } err = e.serviceRegistry.UpdateEndpoints(api.Endpoints{ Name:      service.ID,

What would you like to be : We'd like to have the Prometheus stack () running in our scalability tests. Why is this needed: There are couple of reasons: will simplify writing SLI assertions/tests that are based on metrics not necessarily exported on master. Some examples here are the new In-Cluster Network Programming Latency SLI (exported in kube-proxy) or soon to be implemented RTT SLI where the metric will be exported in regular pods (probes). With Prometheus stack, collecting and aggregating such metrics is very simple. -prometheus provides many useful metrics about the state of the cluster, such metrics might be invaluable when debugging scalability test failures. mimics what customers do in their cluster, making the test setup more realistic.
/sig scalability
/assign
The GCE 5K Node Performance test with enabled Prometheus completed successfully. We can enable Prometheus in all tests.
Looks like kubemark-500 has some issue, test fails often with Kubemark-5000 also failed once with Will disable Prometheus in kubemark, will work on fixing it when I'm back from my vacation.
For the first kind of the failure that kubemark-500 suffered from, i.e. the root cause is Looks like the token for the "prometheus-operator" service account was not ready, as we can see in the logs it was created after the POST to create the servicemonitor was called.
It looks like the second failure (kubemark-5000) was caused by the prometheus-k8s pods not scheduled by the kube-scheduler. This is strange but may be related to other issues we had. Hard to investigate what really happened there, will leave it for now. Will re-enable prometheus tomorrow and will see if it happens again.
Closing, as this is basically done in a minimal version. Other improvements will be tracked in separate issues. /close
Closing this issue. <detailsIn response to : Closing, as this is basically done in a minimal version. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details