Question: framework.ExpectNoError(err) }) <del> /* Create a ClusterIP service with an External IP that is not assigned to an interface. The IP ranges here are reserved for documentation according to [RFC 5737](https://tools.ietf.org/html/rfc5737) Section 3 and should not be used by any host. */ ginkgo.It("should be possible to connect to a service via ExternalIP when the external IP is not assigned to a node", func() { serviceName := "externalip-test" ns := f.Namespace.Name externalIP := "203.0.113.250" if framework.TestContext.ClusterIsIPv6() { externalIP = "2001:DB8::cb00:71fa" } jig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating service " + serviceName + " with type=clusterIP in namespace " + ns) clusterIPService, err := jig.CreateTCPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.ExternalIPs = []string{externalIP} svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolTCP, TargetPort: intstr.FromInt(9376)}, } }) framework.ExpectNoError(err) err = jig.CreateServicePods(2) framework.ExpectNoError(err) execPod := e2epod.CreateExecPodOrFail(cs, ns, "execpod", nil) err = jig.CheckServiceReachability(clusterIPService, execPod) framework.ExpectNoError(err) }) </del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. ginkgo.It("should be able to change the type and ports of a service [Slow] [DisabledForLargeClusters]", func() { // requires cloud load-balancer support

Which jobs are failing: Which test(s) are failing: - Since when has it been failing: Testgrid link: Reason for failure: Nov 4 11:59:41.385: Unexpected error: <*errors.errorString | 0xc00298c3d0: { s: "service is not reachable within 2m0s timeout on endpoint 203.0.113.250:80 over TCP protocol", } service is not reachable within 2m0s timeout on endpoint 203.0.113.250:80 over TCP protocol occurred Anything else we need to know: /priority important-soon /sig network /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Apparently this is related to /cc
if you're busy and want me to take a look tomorrow you can assign that to me and I can deep dive into what's missing/happening :) Happy to help and to understand better the flakiness life
so, the test adds a "fake" external IP that should be nated to the service endpoint It went through the CI, you can check the test here passing more details in the issue I opened explaining the test
ok, this is interesting the test is failing when using container runtime DOCKER https://k8s- the same periodic job that uses container runtime CONTAINERD (same as the PR) is working https://k8s- We need to understand what is different regarding the network in GCE jbos, when using docker as runtime, kubenet??? cc:
one diff between the working jobs with containerd and the failing ones with docker is that the failing ones use this part of the code for reference, kube-proxy adds the NAT rules here for external IPs here
IIRC you are using kubenet for azure jobs, are you using docker as runtime, can you please check if this tests fails there too?
The problem happens on environment that are using the bridge plugin, seems I got wrong the feature will revert until we have the right test.

We use kubenet only for IPv6 and dualstack jobs. Rest of the tests use Azure CNI. Default is docker runtime for all jobs.


Question: - apiGroups: - "" resources: <del> - endpoints verbs: - create - apiGroups: - "" resourceNames: - kube-scheduler resources: - endpoints verbs: - get - update - apiGroups: - "" resources: </del> - nodes verbs: - get

Not , but removed: Looking at this comment: this should be removed once the migration to lease in leader-election was done. Can this be removed already? I mean, the following 2 lines: Security folks internally were asking if we could get rid of this upstream if it is not needed anymore (if this "migration to lease in leader-election" was fully done)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Support for endpoints and configmap based leader election has been removed with this PR: Validation to error out leader elections configuration for resource lock type other than "lease" has been implemented in With this, it looks safe to remove the two lines mentioned in this issue.


Question: o.WaitForDeletion = true o.GracePeriod = 1 } <ins> } else if o.ForceDeletion { fmt.Fprintf(o.ErrOut, "warning: --force is ignored because --grace-period is not 0.n") </ins> } return nil }

IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy


Question: sort.StringSlice(res).Sort() return res } <ins> // Len returns the size of the set. func (s StringSet) Len() int { return len(s) } </ins>

gives but gives If minions were already in etcd from some previous launch of the apiserver/controller things work just fine no matter what the order. So to reproduce you have to clear everything from etcd before you try...
can you take a look at this? Probably just need to retry in background until etcd comes up?
Yep, there is no control loop for static machine config. Adding one should be trivial.
Blocked by Basically, apiserver returns inconsistent response.
I have observed a similar sort of thing, but what it appeared to be is more like the is not listening on it's socket at the time tries to connect to it and it never retries. So my work-around is to run before it like so: What I then observed is that the connection is indeed being refused at first attempt, but works on the second.
what do you see in the logs? I was seeing that it basically fails to connect to once and doesn't retry (although it said that it will try to retry).
should fix it, sorry for the trouble wondering where did you see that "it will try to retry"?
that wasn't the exact wording there... let me see if I can find out what exactly it said.
Ok, so what I'm seeing now is: I hope that PIDs would be a good enough proof to what order services have started up: I could provide some systemd graph dumps, but that would probably be too verbose, PIDs seem clearer. So really don't see how got //apisevercontroller-managerapiserver` didn't open the socket yet...


Question: return &Result{singular: singular, visitor: visitors, sources: b.paths} } <ins> if len(b.resources) != 0 { return &Result{err: fmt.Errorf("resource(s) were provided, but no name, label selector, or --all flag specified")} } </ins> return &Result{err: fmt.Errorf("you must provide one or more resources by argument or filename (%s)", strings.Join(InputExtensions, "|"))} }

When users try to label/annotate multiple resources, we should tell them to add . Current output: Expected output:
and too
Instead of "--all is false", say "multiple resources were provided, but no label selector or --all flag specified"
Description updated based on suggestion
I think "resource(s) were provided, but no name, label selector, or --all flag specified" is better. I will send a pull request for this as my first PR.


Question: p.OnEndpointSlicesSynced() } <ins> func TestFindRemoteSubnetProviderAddress(t *testing.T) { networkInfo, _ := newFakeHNS().getNetworkByName("TestNetwork") pa := networkInfo.findRemoteSubnetProviderAddress(providerAddress) if pa != providerAddress { t.Errorf("%v does not match %v", pa, providerAddress) } pa = networkInfo.findRemoteSubnetProviderAddress(epIpAddressRemote) if pa != providerAddress { t.Errorf("%v does not match %v", pa, providerAddress) } pa = networkInfo.findRemoteSubnetProviderAddress(serviceVip) if len(pa) != 0 { t.Errorf("Provider address is not empty as expected") } } </ins> func makeNSN(namespace, name string) types.NamespacedName { return types.NamespacedName{Namespace: namespace, Name: name} }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Overlay mode, When 2 service has the same backend node IP, the LB was not programmed correctly causing datapath issues. What you expected to happen: Service data path to work properly How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /sig windows These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: // // These are the options which would have to be used: // o := utils.PatchCSIOptions{ <del> // 	OldDriverName:            "com.google.csi.gcepd", // 	NewDriverName:            "com.google.csi.gcepd-" + g.f.UniqueName, </del> <ins> // 	OldDriverName:            g.driverInfo.Name, // 	NewDriverName:            GetUniqueDriverName(g), </ins> // 	DriverContainerName:      "gce-driver", // 	ProvisionerContainerName: "csi-external-provisioner", // }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: if spec.IsKubeletExpandable() { // for kubelet expandable volumes, return a noop plugin that // returns success for expand on the controller <del> klog.Warningf("FindExpandablePluginBySpec(%s) -> returning noopExpandableVolumePluginInstance", spec.Name()) </del> <ins> klog.V(4).Infof("FindExpandablePluginBySpec(%s) -> returning noopExpandableVolumePluginInstance", spec.Name()) </ins> return &noopExpandableVolumePluginInstance{spec}, nil } <del> klog.Warningf("FindExpandablePluginBySpec(%s) -> err:%v", spec.Name(), err) </del> <ins> klog.V(4).Infof("FindExpandablePluginBySpec(%s) -> err:%v", spec.Name(), err) </ins> return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In an otherwise-stable bare-metal setup, with only LocalVolume and NFS storage, and without any volume mount activity, I get 80,000+ of these warning messages per day: What you expected to happen: No warnings; and if there is a reason to generate these warnings, more info about what operation is being attempted should be to the warnings message. How to reproduce it (as minimally and precisely as possible): I don't know. This issue emerged after I about 40 services and 300 volumes on a 3-worker cluster. Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.0Cloud provider or hardware configuration: Bare-metal OS (e.g. from /etc/os-release): Ubuntu 18.04.1 bionic Kernel (e.g. ): 4.15.0-42-generic Install tools: kubeadm Others:
/sig storage


Question: "doc.go", "register.go", "types.go", <ins> "well_known_labels.go", </ins> "zz_generated.deepcopy.go", ], tags = ["automanaged"],

used to show masters, but as of it no longer does. This is not in accordance with our deprecation policy I believe. cc
That commit/PR is not relevant as there are no supported users/deployment methods that are using label. kubeadm v1.4 or v1.5 was the last. What you're really looking for is who/what removed printing things based on the label...
No blame on you The regression is that we have stopped printing masters in . There may be more to it that a simple revert of , but we should probably start there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed in recent versions


Question: Bare-metal     | custom       | Ubuntu | _none_      | [docs](../../docs/getting-started-guides/ubuntu_single_node.md) | Community (@jainvipin)       | Local          |              |        | _none_      | [docs](../../docs/getting-started-guides/locally.md)   | Community (@preillyme)                     | libvirt/KVM    | CoreOS       | CoreOS | libvirt/KVM | [docs](../../docs/getting-started-guides/libvirt-coreos.md) | Community (@lhuard1A)   | <del> Ovirt          |              |        |             | [docs](../../docs/getting-started-guides/ovirt.md)     | Inactive (@simon3z)          | </del> <ins> oVirt          |              |        |             | [docs](../../docs/getting-started-guides/ovirt.md)     | Community (@simon3z)         | </ins> *Note*: The above table is ordered by version test/used in notes followed by support level.

There was once a but it is gone. So I propose deleting , , and , and references to . I'll let this sit for a few days for response before deleting.


Question: # Create a copy of mypod named my-debugger with my-container's image changed to busybox kubectl alpha debug mypod --image=busybox --container=my-container --copy-to=my-debugger -- sleep 1d <ins> # Create an interactive debugging session on a node and immediately attach to it. # The container will run in the host namespaces and the host's filesystem will be mounted at /host kubectl alpha debug node/mynode -it --image=busybox </ins> `)) )

This issue tracks implementing node debugging with privileged containers for , as described in the This feature allows administrators to start a privileged container on a node in host namespaces rather than having to SSH to the node. For example: /kind feature /sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } } <ins> func TestPatchNoop(t *testing.T) { _, svc, _ := testData() getObject := &svc.Items[0] patchObject := &svc.Items[0] f, tf, codec, _ := cmdtesting.NewAPIFactory() tf.UnstructuredClient = &fake.RESTClient{ APIRegistry:          api.Registry, NegotiatedSerializer: unstructuredSerializer, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { switch p, m := req.URL.Path, req.Method; { case p == "/namespaces/test/services/frontend" && m == "PATCH": return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: objBody(codec, patchObject)}, nil case p == "/namespaces/test/services/frontend" && m == "GET": return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: objBody(codec, getObject)}, nil default: t.Fatalf("unexpected request: %#vn%#v", req.URL, req) return nil, nil } }), } tf.Namespace = "test" // No-op { buf := bytes.NewBuffer([]byte{}) cmd := NewCmdPatch(f, buf) cmd.Flags().Set("namespace", "test") cmd.Flags().Set("patch", `{}`) cmd.Run(cmd, []string{"services", "frontend"}) if buf.String() != "service "baz" not patchedn" { t.Errorf("unexpected output: %s", buf.String()) } } // Patched { copied, _ := api.Scheme.DeepCopy(patchObject) patchObject = copied.(*api.Service) if patchObject.Annotations == nil { patchObject.Annotations = map[string]string{} } patchObject.Annotations["foo"] = "bar" buf := bytes.NewBuffer([]byte{}) cmd := NewCmdPatch(f, buf) cmd.Flags().Set("namespace", "test") cmd.Flags().Set("patch", `{"metadata":{"annotations":{"foo":"bar"}}}`) cmd.Run(cmd, []string{"services", "frontend"}) if buf.String() != "service "baz" patchedn" { t.Errorf("unexpected output: %s", buf.String()) } } } </ins> func TestPatchObjectFromFileOutput(t *testing.T) { _, svc, _ := testData()

The bug is introduced in The oldData, newData in seem always to be the same. So it always showing . After digging a bit, I found that: the place where we save the may be wrong. sometimes oldData and newData are different because of different s, even though the patch is . cc:
cc
info.Object is updated by the info.Refresh call. We should compute old data prior to refreshing the info bug was introduced in
I think the second bug is because using read from stream has less infomation than when you fetch from server, so but didn't


Question: ) fp.syncProxyRules() <ins> // Expect 2 services and 1 destination epVS := &netlinktest.ExpectedVirtualServer{ VSNum: 2, IP: svcLBIP, Port: uint16(svcNodePort), Protocol: string(api.ProtocolTCP), RS: []netlinktest.ExpectedRealServer{{ IP: epIP, Port: uint16(svcPort), }}} checkIPVS(t, fp, epVS) // check ipSet rules epIPSet := netlinktest.ExpectedIPSet{ kubeLoadBalancerSet: {{ IP:       svcLBIP, Port:     svcPort, Protocol: strings.ToLower(string(api.ProtocolTCP)), SetType:  utilipset.HashIPPort, }}, } checkIPSet(t, fp, epIPSet) // Check iptables chain and rules epIpt := netlinktest.ExpectedIptablesChain{ string(kubeServicesChain): {{ JumpChain: string(KubeLoadBalancerChain), MatchSet: kubeLoadBalancerSet, }}, string(kubeLoadBalancerSet): {{ JumpChain: string(KubeMarkMasqChain), MatchSet: "", }}, } checkIptables(t, ipt, epIpt) </ins> } func TestOnlyLocalNodePorts(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST? /kind bug What happened: In ipvs proxy mode, most service functions like "nodeport/loadbalance" is realized by the association of iptables/ipset/ipvs, yet most ipvs proxier ut only verified ipvs rules, which can't ensure service is functional. What you expected to happen: Check both ipvs rules and iptables chains/rules, ipset entries in ipvsproxier ut.
/sig network /area ipvs /assign
What's the current UT coverage number?
Current is 50.2%.
So, can we achieve 70%?
Yeah, we can do that, and that's definitely necessary.


Question: volumeSpec := controllervolumetesting.GetTestVolumeSpec(string(volumeName), volumeName) nodeName := k8stypes.NodeName("node-name") dsw.AddNode(nodeName, false /*keepTerminatedPodVolumes*/) volumeExists := dsw.VolumeExists(volumeName, nodeName) if volumeExists { t.Fatalf(

A/D controller tries to detach volumes that are still mounted after 6 minutes. The conditions to trigger the detach are: I.e. all containers are finished. But that does not guarantee that volumes are unmounted. When a CSI driver can't unmount a volume for 6 minutes, the volume is force detached by A/D controller, corrupting the volume and leaving volume mounts on the node in a bad state. Note that the node is perfectly healthy, it just can't unmount a volume from whatever reason. A/D controller detaches volumes after a Pod is deleted from the API server, i.e. the volumes are unmounted or someone force-deleted the pods. Very artificial reproducer: a Pod with an attachable CSI volume. the node, make the volume mount busy, e.g.: Every unmount of the volume fails and kubelet/the CSI driver periodically retry. the pod 6 minutes, see the volume is detached without unmounting first. KCM logs: Note that the Pod is still in the API server: And the node is healthy: Note that some CSI drivers (e.g. the cloud ones) do not allow detaching a volume while it's still mounted. Not all CSI driver have such possibility and they rely on Kubernetes not to call ControllerUnpublish before NodeUnpublish / NodeUnstage succeeds. <detailsToday's master </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-longterm
Right now, a volume is detached from a node when: (all pods that use the volume are deleted from the API server OR all their container are stopped) AND (the volume is unmounted OR 6 minutes passed). Potential solution 1: Issue detach of a mounted volume only when all pods are force-deleted from the API server.

More formally: Detach when [(All pods that use the volume are deleted from the API server OR all their container are stopped) AND the volume is unmounted] OR [all pods that use the volume are deleted from the API server AND 6 minutes passed]. Is this change subtle enough that it does not break existing behavior (e.g. tooling on top of Kubernetes)?
Potential solution 2: Add a new flag to CSIDriver, say . We would need to extend internal Detach() call to include a flag if the volume to detach is mounted or not (A/D controller knows that) and the CSI volume plugin would check CSIDriver and potentially reject detach of a mounted volume. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 3: As a variant, we could add a cmdline option to the external-attacher, like . When that cmdline option is specified, the external attacher would check , if the volume to detach is still mounted, and refuse such detach (with a reasonable error message). CSI driver vendors could opt-in into this behavior. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 4: As a variant, we could add to CSIDriver, where each driver could put their own timeout before force-detaching a volume, with an option to use infinity.
Note to self: check , it might help on the CSI side.
Solution 3 will not work well with ungraceful node shutdown - attach/detach controller does not watch pods and force-deleting pods is the main point of ungraceful node shutdown KEP. /me will try if solution 1 would work when NodeStage times out (is Pod deleted from the API server before or after NodeUnstage is called?)
was right, a pod is deleted (without force) from the API server after NodeUnpublish succeeds, it does not wait for NodeUnstage to finish. Therefore we cannot distinguish if a pod was force-deleted (and we should start 6 minute timer) or pod was deleted without any force and only NodeUnstage is taking more time than expected. Solution 1 is then not possible.
Now we are down to solution 2 and 4.

For solution 2, if driver does not have a way to check whether volume is mounted or not, the behavior will not change and we still force detach after 6 mins. And it might happen in the middle of unstage ..
Summing up yesterday's call: A/D controller should check if a node is unhealthy before force detaching volumes from it. It should never force detach a volume from a healthy node, not even after 6 minutes. Question what healthy means is open and we should talk to sig-node. is a working assumption.
I agree with both of these. I'd just emphasize that that particular definition of unhealthy definitely includes perfectly healthy nodes in a network partition. This change is no worse than the status quo ante for those nodes, and a definite improvement on the healthy nodes, so I wholeheartedly support it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign


Question: // Expect EndpointSlice resource to be created. var endpointSlice discoveryv1beta1.EndpointSlice <del> if err := wait.PollImmediate(2*time.Second, 12*time.Second, func() (bool, error) { </del> <ins> if err := wait.PollImmediate(2*time.Second, wait.ForeverTestTimeout, func() (bool, error) { </ins> endpointSliceList, err := cs.DiscoveryV1beta1().EndpointSlices(svc.Namespace).List(context.TODO(), metav1.ListOptions{ LabelSelector: "kubernetes.io/service-name=" + svc.Name, })

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ), }, expected: &DeviceRunContainerOptions{ <del> Annotations: []kubecontainer.Annotation{ {Name: "cdi.k8s.io/devicemanager_pod-container", Value: "vendor1.com/class1=device1,vendor2.com/class2=device2"}, }, </del> CDIDevices: []kubecontainer.CDIDevice{ {Name: "vendor1.com/class1=device1"}, {Name: "vendor2.com/class2=device2"},

Kubelet should stop using annotations to pass CDI device IDs to CRI runtimes. CDIDevices CRI field should be used for this purpose. This should be done when Kubelet that doesn't support CRI field reaches EOL and two major CRI runtimes support CRI field. Here is a PR comment that gives more details about this: Kubelet started using CDIDevices CRI field in . Previous version 1.27 will reach EOL .CRI-O started using it in . Previous version 1.27 will reach EOL together with Kubernetes 1.27.Containerd started using it in . Version 1.7 will reach EOL in a 6 months after Containerd 2.0 release Previous version 1.6 doesn't support CDI injections and should not be considered. So, safer option would be to wait until Containerd 1.7 reaches EOL. However it's relatively safe to do it earlier when 1.7.0 and 1.7.1 have little chances to be in real use. Containerd v1.7.0 and v1.7.1 is not used by the Linux distributions from , so it's safe to remove annotations support from the Kubelet in 1.31 in my opinion.
/sig node
/triage accepted /priority important-longterm
/assign


Question: if len(items) == 0 { lruItemObj, ok := l.liveLookupCache.Get(a.GetNamespace()) if !ok || lruItemObj.(liveLookupEntry).expiry.Before(time.Now()) { <del> // TODO: If there are multiple operations at the same time and cache has just expired, // this may cause multiple List operations being issued at the same time. // If there is already in-flight List() for a given namespace, we should wait until // it is finished and cache is updated instead of doing the same, also to avoid // throttling - see #22422 for details. liveList, err := l.client.CoreV1().LimitRanges(a.GetNamespace()).List(context.TODO(), metav1.ListOptions{}) </del> <ins> // Fixed: #22422 // use singleflight to alleviate simultaneous calls to lruItemObj, err, _ = l.group.Do(a.GetNamespace(), func() (interface{}, error) { liveList, err := l.client.CoreV1().LimitRanges(a.GetNamespace()).List(context.TODO(), metav1.ListOptions{}) if err != nil { return nil, admission.NewForbidden(a, err) } newEntry := liveLookupEntry{expiry: time.Now().Add(l.liveTTL)} for i := range liveList.Items { newEntry.items = append(newEntry.items, &liveList.Items[i]) } l.liveLookupCache.Add(a.GetNamespace(), newEntry) return newEntry, nil }) </ins> if err != nil { <del> return nil, admission.NewForbidden(a, err) } newEntry := liveLookupEntry{expiry: time.Now().Add(l.liveTTL)} for i := range liveList.Items { newEntry.items = append(newEntry.items, &liveList.Items[i]) </del> <ins> return nil, err </ins> } <del> l.liveLookupCache.Add(a.GetNamespace(), newEntry) lruItemObj = newEntry </del> } lruEntry := lruItemObj.(liveLookupEntry)

Recently we observed high latencies on POST pods on our scalability tests. For more context see: It appeared to be throttling related, e.g.: As an example look at resource quota admission controller. In there was some fallback to use lru cache in case of on objects. However, not what happens if we are sending multiple POST pods at the same time to apiserver. What happens in Admit() if there are no results in Indexer, we list ResourceQuotas. But if there are multiple calls at the same time, there can be multiple LISTs before the cache is updated: But this can cause throttling of these requests. What we should do is: if there is List() in flight, we should wait until it is finished and lookupCache is updated.
- thanks for the analysis. - fyi, this is an interesting side-effect of the LRU cache pattern, not sure if we have adopted it in other places as well, but good to keep in mind.
To clarify, it was adopted in and admission plug-ins in separate PRs.
- it took me some time, but was useful experience. Thanks for assigning yourself.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ) fi <del> kube::build::copy_output </del> kube::release::package_tarballs kube::release::package_hyperkube

It fails with: cc
Looking at the generated I see that FEDERATIONIMAGETAG is "" which is resulting in this error. I see that before , FEDERATIONIMAGETAG was being set to but that docker tag file does not exist for hyperkube. I see the following exist in my : but does not exist there. Trying to figure out why.
cc
make sure you are running the scripts with env var exported. That is what controls whether gets created.
With , we are not generating the federation-apiserver binary and instead using the hyperkube binary.
sorry, i copied the wrong line. The droid you're looking for: File conditionally written during process File consumed when templating federation control plane yaml
Right. My is also empty. I am trying to run federation e2e tests. The command I am running is: which is what I have always used. I guess something changed in
that should definitely work. Please and then run the step, capture the output and email to me? I was literally doing exactly this process for another PR today, and I verified the image tag file was getting written correctly as well.
I figured out this is happening because after make clean or a fresh clone, kubectl will not be available when we are trying to get the semanticimagetag_version() in here.
Awesome! Thanks


Question: } // CleanupLeftovers clean up all ipvs and iptables rules created by ipvs Proxier. <del> func CleanupLeftovers(ipvs utilipvs.Interface, ipt utiliptables.Interface, ipset utilipset.Interface, cleanupIPVS bool) (encounteredError bool) { if cleanupIPVS { // Return immediately when ipvs interface is nil - Probably initialization failed in somewhere. if ipvs == nil { return true } encounteredError = false </del> <ins> func CleanupLeftovers(ipvs utilipvs.Interface, ipt utiliptables.Interface, ipset utilipset.Interface) (encounteredError bool) { // Clear all ipvs rules if ipvs != nil { </ins> err := ipvs.Flush() if err != nil { klog.Errorf("Error flushing IPVS rules: %v", err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy don’t clean ipvs rules with . What you expected to happen: kube-proxy clean ipvs rules with by default. How to reproduce it (as minimally and precisely as possible): Run kube-proxy in ipvs mode, and clean rules with . Anything else we need to know?: Based on , should cleanup all rules by default, including ipvs rules. And also retains functionality. Environment: Kubernetes version (use ): v1.20.0-bate.0 Cloud provider or hardware configuration: OS (e.g: ): centos7 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
it seems that this flag entered deprecation, so I think that recovering functionaly is not the right approach you were involved in the discussion with this flag in , can you take a look at this?
thanks for your reply. The was in , to avoid removing IPVS rules which are not created by kube-proxy. According to , It seems that deleting ipvs rules by default is expected, but it is not so. I think that keeping --cleanup-ipvs is meaningful, unless kube-proxy can only delete the IPVS rules created by itself.
/assign
sorr for the late answer , but the flag must have the same functionality than , actually I can see that it has implemented. In case that the is not working we should fix it, but not recover the deprecated flag
Agree
sorry if I'm missing something. I just try to make the function of as described, not to recover the flag. The problem I see is: The can't flush IPVS. Because the IPVS flush process is still controlled by . Beforce , it was assigned by flag, but now it is not assigned, always false. currently does not have any functionality, just stored by . So, should we keep the function of temporarily by assigning it to ,or just make always flushing IPVS regardless of the ?

I prefer the former because the flag is not completely abandoned :)
it seems that is what the flag description says, you have to use to flush the ipvs entries if that is the case it seems that we are ok, right? no changes needed
In fact, can't flush ipvs entries. The reason is what I described above.
I´m sorry, I think I got it know, the flag is not exposed, but the option is there and initialized to true, this is what I see so, it seems that flush IPVS by default
yes, should flush IPVS by default, but it doesn’t actually
... and finally I got it, so sorry :bow: the ProxyServer is not being instantiated with the flag, despite is set :facepalm: you are more familiar with the IPVS implementation than me, I personally think that, if the flag deprecation has started, we should make that always flush IPVS and totally remove ... it seems aligned with the other proxy implementations
/triage accepted
Actually, after doing some code archaeology, this was the expectation per And the flag clearly mentions that --cleanup must flush IPVS entries too I think that we can go ahead and remove the --cleanup-ipvs flag and make --cleanup to flush ipvs too, right now is a mess and is not working at all , are you going to submit a PR?
thanks for digging up the history on this. That lines up with my recollection and the KEP around kube-proxy's clean up logic
OK, I will create a PR to remove the flag


Question: }, []metav1beta1.TableRow{{Cells: []interface{}{"test5", "1/2", "podReason", 6, "<unknown>"}}}, }, <ins> { // Test pod has 2 containers, one is running and the other is completed. api.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "test6"}, Spec:       api.PodSpec{Containers: make([]api.Container, 2)}, Status: api.PodStatus{ Phase:  "Running", Reason: "", ContainerStatuses: []api.ContainerStatus{ {Ready: true, RestartCount: 3, State: api.ContainerState{Terminated: &api.ContainerStateTerminated{Reason: "Completed", ExitCode: 0}}}, {Ready: true, RestartCount: 3, State: api.ContainerState{Running: &api.ContainerStateRunning{}}}, }, }, }, []metav1beta1.TableRow{{Cells: []interface{}{"test6", "1/2", "Running", 6, "<unknown>"}}}, }, </ins> } for i, test := range tests {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When one pod is running and has two Containers. Container 1 exits with success and container 2 is running. gives correct info - the pod status is . However, , the pod status is . What you expected to happen: , the pod status is . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
i can not see your pod named nginx in your above code
, updated the typo.
/sig node /remove-sig api-machinery


Question: . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "k8s.io/api/core/v1" <ins> policy "k8s.io/api/policy/v1beta1" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> clientset "k8s.io/client-go/kubernetes" v1core "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/kubernetes/pkg/api/testapi"

When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .


Question: Name: makeContainerName(sandboxConfig, config), Config: &dockercontainer.Config{ // TODO: set User. <del> Hostname:   sandboxConfig.GetHostname(), </del> Entrypoint: dockerstrslice.StrSlice(config.GetCommand()), Cmd:        dockerstrslice.StrSlice(config.GetArgs()), Env:        generateEnvList(config.GetEnvs()),

umbrella issue: kuberuntime uses to ]parse the version string of the api version]() from the runtime. When using with docker, it complains that Maybe we should just enforce that the api version returned by the runtime should be semver compatible. /cc
This is mentioned in . :) For now, we can add a after the docker apiversion to make it semver compatible.
Thanks for the reminder! We should make it clear in the API. This blocks kubelet from starting up, so we should fix it first.
I've checked the code, currently is only used in when checking the docker version compatibility. It is not exposed to the user by any means, so it should be fine to make some change ourselves, such as adding a . However, if we want to expose this to the user via in the future, it may make user confusing.


Question: ginkgo.By(fmt.Sprintf("Checking that %d pod created and status is failed", backoff+1)) pods, err := e2ejob.GetJobPods(f.ClientSet, f.Namespace.Name, job.Name) framework.ExpectNoError(err, "failed to get PodList for job %s in namespace: %s", job.Name, f.Namespace.Name) <del> // gomega.Expect(pods.Items).To(gomega.HaveLen(backoff + 1)) // due to NumRequeus not being stable enough, especially with failed status // updates we need to allow more than backoff+1 // TODO revert this back to above when https://github.com/kubernetes/kubernetes/issues/64787 gets fixed if len(pods.Items) < backoff+1 { framework.Failf("Not enough pod created expected at least %d, got %#v", backoff+1, pods.Items) } </del> <ins> gomega.Expect(pods.Items).To(gomega.HaveLen(backoff + 1)) </ins> for _, pod := range pods.Items { framework.ExpectEqual(pod.Status.Phase, v1.PodFailed) }

What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): $oc create job -f $ oc get job jobname -ojson | jq .status { "active": 1, "failed": 1, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 2, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 3, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 4, "startTime": "2020-07-30T06:45:37Z" } { "conditions": [ { "lastProbeTime": "2020-07-30T06:53:10Z", "lastTransitionTime": "2020-07-30T06:53:10Z", "message": "Job has reached the specified backoff limit", "reason": "BackoffLimitExceeded", "status": "True", "type": "Failed" } ], "failed": 5, "startTime": "2020-07-30T06:45:37Z" } $ oc get job jobname -ojson | jq .spec.backoffLimit 6 Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: 4.5.3 Server Version: 4.4.13 Kubernetes Version: v1.17.1+ Cloud provider or hardware configuration: OS (e.g: ): NAME="Red Hat Enterprise Linux CoreOS" VERSIONID="4.4" OPENSHIFTVERSION="4.4" RHELVERSION="8.2" ID="rhcos" IDLIKE="rhel fedora" ANSICOLOR="0;31" CPENAME="cpe:/o:redhat:enterpriselinux:8::coreos" HOMEURL="" BUGREPORTURL="" REDHATBUGZILLAPRODUCT="OpenShift Container Platform" REDHATBUGZILLAPRODUCTVERSION="4.4" REDHATSUPPORTPRODUCT="OpenShift Container Platform" REDHATSUPPORTPRODUCTVERSION="4.4" OSTREEVERSION='44.82.-0' Kernel (e.g.

): Linux ip-10-0-192-232 4.18.0-193.13.2.el82.x8664 SMP Mon Jul 13 23:17:28 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Others: apiVersion: batch/v1 kind: Job metadata: name: sre-build-test-fail namespace: openshift-build-test spec: backoffLimit: 6 completions: 1 parallelism: 1 template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: matchExpressions: key: node- operator: In values: infra containers: command: /bin/bash -c | # ensure we fail if something exits non-zero set -o errexit set -o nounset set -o pipefail exit 1 imagePullPolicy: Always name: sre-build-test restartPolicy: Never terminationGracePeriodSeconds: 30 tolerations: effect: NoSchedule key: node- operator: Equal value: infra
/sig apps
not sure but that is the best I can think of at this time
i have noticed this problem also, and sometimes, it seems to work normally.
so I found out that in the test is so this can cause the issue.. it has been there from the beginning of the codes inception.. WDYT?
/assign
Duplicate of . /close
Closing this issue. <detailsIn response to : Duplicate of . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "fmt" "math/rand" <ins> "net" "os" </ins> "reflect" "runtime" "sync"

No response No response /sig windows /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig windows
i think this problem can be fixed when it imports os/net in Am i right?
I also found some other unit test bugs in CI, such as running linux-related tests on windows. I will try to fix them. /assign
/cc
Thanks for opening the issue! I didn't notice the failure through the testgrid dashboard. I've sent a PR here: . While fixing this issue, I noticed there's one unit test that was failing, so that PR also addresses that as well.
Yep, the imports you mentioned are needed. Keep in mind that imports are in alphabetical order. But in addition to that, there was another failing unit test which was masked by this import errors.


Question: data = ["//staging/src/k8s.io/kubectl/testdata"], embed = [":go_default_library"], deps = [ <del> "//staging/src/k8s.io/api/apps/v1:go_default_library", </del> "//staging/src/k8s.io/api/batch/v1:go_default_library", "//staging/src/k8s.io/api/batch/v1beta1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of Job created by using command will be . This apiVersion is wrong. What you expected to happen: Set the to How to reproduce it (as minimally and precisely as possible): CronJob manifest: <details</detailsApply CronJob and execute command Anything else we need to know?: Environment: Kubernetes version (use ):
/sig cli
This was introduced by . /assign
GitHub didn't allow me to assign the following users: odino. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This was introduced by . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I created fix pull request:


Question: } } <ins> func TestNonRacyShutdown(t *testing.T) { // Attempt to simulate previously racy conditions, and ensure that no race // occurs: Nominally, calling "Eventf" *followed by* shutdown from the same // thread should be a safe operation, but it's not if we launch recorder.Action // in a goroutine. caster := NewBroadcasterForTests(0) clock := clock.NewFakeClock(time.Now()) recorder := recorderWithFakeClock(v1.EventSource{Component: "eventTest"}, caster, clock) var wg sync.WaitGroup wg.Add(100) for i := 0; i < 100; i++ { go func() { defer wg.Done() recorder.Eventf(&v1.ObjectReference{}, v1.EventTypeNormal, "Started", "blah") }() } wg.Wait() caster.Shutdown() } </ins> func TestEventf(t *testing.T) { testPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{

The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?


Question: log-wrap 'CreateDirs' create-dirs log-wrap 'EnsureLocalSSDs' ensure-local-ssds log-wrap 'SetupKubeletDir' setup-kubelet-dir <ins> log-wrap 'SetupJournald' setup-journald </ins> log-wrap 'SetupLogrotate' setup-logrotate if [[ "${KUBERNETES_MASTER:-}" == "true" ]]; then log-wrap 'MountMasterPD' mount-master-pd

Example using a random PR for , take a look at: in this set of logs, at least 4 files are empty kube-container-runtime- kube-node- kube- kubelet- The reason tracked down using was that the journald configuration was defaulting to something that wipes the logs quickly!
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // checkHostPortConflicts checks for colliding Port.HostPort values across // a slice of containers. func checkHostPortConflicts(containers []api.Container) errs.ValidationErrorList { <del> allPorts := map[int]bool{} return AccumulateUniquePorts(containers, allPorts, func(p *api.ContainerPort) int { return p.HostPort }) </del> <ins> allPorts := util.StringSet{} return AccumulateUniqueHostPorts(containers, &allPorts) </ins> } func validateExecAction(exec *api.ExecAction) errs.ValidationErrorList {

Pods which use to bind both TCP and UDP sockets to the same port number are rejected with the error: An example pod follows.
Looks like we don't check protocol
The issue was discussed before in , which was closed due to it being low priority.
Hmm, that's 2 pods and the scheduler, this is 2 containers in the same pod and validation at create time. Maybe we should just be more clear about where we stand with hostPort.
Posting here what I posted in the other bug: This can't be worked around when running NFS from a pod. Linux does not allow the NFS port (2049) to be changed, and the Portmap port (111) also cannot be changed. Additionally, you can't separate the UDP from the TCP ports.
Why does NFS need host ports? , Russell Harmon wrote:
Fix is in , Tim Hockin wrote:


Question: package azure import ( <del> "fmt" </del> "net/http" "strings"

What happened: made sure that API calls to Azure that return a HTTP code 200 with an embedded error are retried. Nevertheless, we observe a series of errors during the Azure Disk detach backoff while calling , and the detach operation is immediately cancelled. Happy scenario: the disk is to be re-attached to the same VM. The reconciler's will update the VMSS instance and the previous detach error will be masked. It takes 15 min for the detach call to timeout but the disk will eventually get re-attached after several VMSS updates (). Not-so-happy scenario: the disk is to be re-attached to another VM. The same errors as in will be thrown indefinitely. controller-manager's logs: What you expected to happen: The backoff succeeds and the failed detach operation gets retried. How to reproduce it: A single Pod with a single Azure Disk. This seems quite hard to reproduce, but it is happening consistently in one of our customer's resource group (eastus). Anything else we need to know?: Environment: Kubernetes version: branch (kubernetes/kubernetes Cloud provider: Azure (via AKS-Engine) OS: Ubuntu 16.04.6 LTS Kernel: 4.15.0-1042-azure Install tools: Network plugin and version: Others: /sig azure paging
/assign


Question: } type dummyStorage struct { <ins> sync.RWMutex </ins> err error }

As Jordan pointed out in: watchcache treats RV=0 and RV="" the same way, even though it's not what we document: This should be fixed. /kind bug /sig api-machinery /priority important-soon
/assign
/triage accepted


Question: // schedulable. Won't have any effect if register-node is false. // DEPRECATED: use registerWithTaints instead RegisterSchedulable bool <del> // RequireKubeConfig is deprecated! A valid KubeConfig is now required if --kubeconfig is provided. RequireKubeConfig bool </del> // nonMasqueradeCIDR configures masquerading: traffic to IPs outside this range will use IP masquerade. NonMasqueradeCIDR string // This flag, if set, instructs the kubelet to keep volumes from terminated pods mounted to the node.

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: fs.BoolVar(&o.CleanupIPVS, "cleanup-ipvs", o.CleanupIPVS, "If true and --cleanup is specified, kube-proxy will also flush IPVS rules, in addition to normal cleanup.") fs.MarkDeprecated("cleanup-ipvs", "In a future release, running --cleanup will always flush IPVS rules") <del> fs.Var(utilflag.IPVar{Val: &o.config.BindAddress}, "bind-address", "The IP address for the proxy server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.Var(utilflag.IPVar{Val: &o.config.HealthzBindAddress}, "healthz-bind-address", "The IP address for the health check server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.Var(utilflag.IPVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address for the metrics server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") </del> <ins> fs.Var(utilflag.IPVar{Val: &o.config.BindAddress}, "bind-address", "The IP address for the proxy server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces)") fs.Var(utilflag.IPPortVar{Val: &o.config.HealthzBindAddress}, "healthz-bind-address", "The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '[::]:10256' for all IPv6 interfaces). Set empty to disable.") fs.Var(utilflag.IPPortVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable.") </ins> fs.Var(utilflag.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.") fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs'. If blank, use the best-available proxy (currently iptables).  If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.") fs.Var(cliflag.NewMapStringBool(&o.config.FeatureGates), "feature-gates", "A set of key=value pairs that describe feature gates for alpha/experimental features. "+ "Options are:n"+strings.Join(utilfeature.DefaultFeatureGate.KnownFeatures(), "n")) fs.Int32Var(&o.healthzPort, "healthz-port", o.healthzPort, "The port to bind the health check server. Use 0 to disable.") <ins> fs.MarkDeprecated("healthz-port", "This flag is deprecated and will be removed in a future release. Please use --healthz-bind-address instead.") </ins> fs.Int32Var(&o.metricsPort, "metrics-port", o.metricsPort, "The port to bind the metrics server. Use 0 to disable.") <ins> fs.MarkDeprecated("metrics-port", "This flag is deprecated and will be removed in a future release. Please use --metrics-bind-address instead.") </ins> fs.Int32Var(o.config.OOMScoreAdj, "oom-score-adj", utilpointer.Int32PtrDerefOr(o.config.OOMScoreAdj, int32(qos.KubeProxyOOMScoreAdj)), "The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]") fs.Int32Var(o.config.IPTables.MasqueradeBit, "iptables-masquerade-bit", utilpointer.Int32PtrDerefOr(o.config.IPTables.MasqueradeBit, 14), "If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with.  Must be within the range [0, 31].") fs.Int32Var(o.config.Conntrack.MaxPerCore, "conntrack-max-per-core", *o.config.Conntrack.MaxPerCore,

What would you like to be amended: lists: I'd like the default for that command line option to actually be , so that the documentation looks right. Internally, the behavior (listen on unless sets a different bind address) should stay the same even if the implementation is updated. Why is this needed: This was originally reported as a website defect (and proposed fix): /kind cleanup /sig network /sig docs /priority backlog
/remove-kind feature
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
should be changed as well
According to Maybe we should deprecate and firstly. If this is the right direction, I am willing to send a PR for this :-)
:thinking: it seems that those flags that allow to specify the ports were "recently" (1.14) I think that is the best person to define the strategy regarding configuration flags, config file, ...


Question: volumeMounts: - name: config mountPath: /etc/config <ins> nodeSelector: beta.kubernetes.io/masq-agent-ds-ready: "true" </ins> volumes: - name: config configMap:

https://k8s- is failing to run the follow up e2es because the ip-masq-agent daemonset is scheduling pods that get stuck in a pending state on 1.5 nodes. This short-circuits the logic for the skewed e2es, and they never run. from https://k8s-
So I see that in 1.7.0 we start setting the non-masq-cidr to 0/0 gce/config-"0.0.0.0/0" We set the addon if non-masq-cidr is 0/0 if [[ "${NONMASQUERADECIDR:-}" == "0.0.0.0/0" ]]; then setup-addon-manifests "addons" "ip-masq-agent" fi Also the agent cannot run on nodes prior to 1.7 so this seems to be correct. Once the nodes are upgraded to 1.7 then the ip-masq agent should be running and should get out of the pending state. I think this is just a general problem with the way addons are handled. I may want to enable a new addon when a master is upgraded to 1.7 by default. That addon may be a DaemonSet that wants to run on nodes meeting a certain criteria. That criteria may not be met in 1.5 but may be met when they are updated to 1.7. A few things that come to mind to fix this: -Fix the test's expectations to allow this until the nodes are updated to 1.7. -rollback the PR that made this the default: -set NONMASQUERADECIDR=10.0.0.0/8 in the test properties so that the ip-masq-agent doesn't get .
It seems like the most conservative way forward is to revert the PR (). Customers who want to use the feature can set the variable () during to be able to get the feature. What do you guys think?
I think I asked the similar question related to ip-masq-agent, but my question was dropped. At this stage, I think reverting the preferred fix.

ip-masq-agent was a daemonset introduced in 1.7 which I questioned at The problem is that after master is upgraded to 1.7, addons is trying to schedule ip-masq-agent daemonset to every node no matter which Kubelet version the node is on. The problem is that ip-masq-agent can only run on node with 1.7 release.
Even if we fix the test it doesn't fix GKE or any other environment where the master is upgraded to 1.7 and the nodes are < 1.7. I think the real problem here is that there is no distinction between staging an addon and enabling an addon. Essentially staging = running. So we end up adding labels to nodes as a way of keeping the DaemonSet disabled until those labels are to nodes. While that works fine for addons that are DaemonDets, it doesn't work for deployments and replica sets. I feel like this is going to happen again regardless of what we do here with ip-masq-agent. It also creates a secondary step. Who applies those addons? What if they forget? what if it's deleted?
Fixed by (the formatting wasn't quite right in the IR to auto-close this issue).
got reverted as it broke tests.


Question: expected []string }{ { <del> name: "custom certs dir", </del> <ins> name: "custom certs dir for v1.12.0-beta.2", </ins> cfg: &kubeadmapi.ClusterConfiguration{ CertificatesDir:   testCertsDir, <del> KubernetesVersion: "v1.7.0", </del> <ins> KubernetesVersion: "v1.12.0-beta.2", </ins> }, expected: []string{ "kube-controller-manager",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create an old version k8s cluster, the controller manager pod can not work well. The command like this : "kubeadm init --kubernetes-version=v1.11.2" The error of kube controller manager pod like this : What you expected to happen: The controller manager pod can work well. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig contributor-experience /area kubeadm
Please, use when filing kubeadm issues. /sig cluster-lifecycle
OK, thank you for reminding me.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/watch" <ins> cachetools "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" </ins> "k8s.io/kubernetes/test/e2e/framework" "github.com/onsi/ginkgo"

By inspection, the "should receive events on concurrent watches in same order" conformance test does two things that are not stable: Starts watches from resourceVersion=0 Relies on watch connections without handling timeouts/restarts This test can fail on clusters that have configmaps autocreated in namespaces already because ordering of watch events for resourceVersion=0 watches is undefined (see ) Watch connections used by conformance tests should use well-defined ordering (starting from resourceVersions other than 0) and handle interruptions properly. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> // WaitForStatusCurrentReplicas waits for the ss.Status.CurrentReplicas to be equal to expectedReplicas func WaitForStatusCurrentReplicas(c clientset.Interface, ss *appsv1.StatefulSet, expectedReplicas int32) { framework.Logf("Waiting for statefulset status.currentReplicas updated to %d", expectedReplicas) ns, name := ss.Namespace, ss.Name pollErr := wait.PollImmediate(StatefulSetPoll, StatefulSetTimeout, func() (bool, error) { ssGet, err := c.AppsV1().StatefulSets(ns).Get(context.TODO(), name, metav1.GetOptions{}) if err != nil { return false, err } if ssGet.Status.ObservedGeneration < ss.Generation { return false, nil } if ssGet.Status.CurrentReplicas != expectedReplicas { framework.Logf("Waiting for stateful set status.currentReplicas to become %d, currently %d", expectedReplicas, ssGet.Status.CurrentReplicas) return false, nil } return true, nil }) if pollErr != nil { framework.Failf("Failed waiting for stateful set status.currentReplicas updated to %d: %v", expectedReplicas, pollErr) } } </ins> // Saturate waits for all Pods in ss to become Running and Ready. func Saturate(ctx context.Context, c clientset.Interface, ss *appsv1.StatefulSet) { var i int32

User cordons a node with a pvc bound to a local persistent volume User deletes a stateful set pod from that node The pod is in pending because the pvc is still bound to a local pv on the cordoned node (expected) The node dies and the pvc is deleted The pod is still stuck in pending because the pvc is missing (bug) The pod now needs to be manually deleted so that the pod+pvc can be recreated After the pvc was deleted, the statefulset controller should have recreated it and bound it to a new node's local pv so that the pending pod could be scheduled. Currently the only way to get the pod out of pending is to manually delete it so that it recreates itself and its pvc We used for the local PVs, but the problem is not limited to this particular provisioner create a stateful set with a pvc that is bound to a local PV use to cordon a node with a local pv delete a statefulset pod from that node delete the pvc from that node the pod is now stuck in pending due to a missing pvc There was a related issue but this only fixed the issue when the order of operations was delete pvc -delete pod. but in this case, it is delete pod -delete pvc <details</details<detailsTested on aws and gcp </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
This issue was fixed by The new issues look different. /close
Closing this issue. <detailsIn response to : This issue was fixed by The new issues look different. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper) <ins> if utilfeature.DefaultFeatureGate.Enabled(genericfeatures.APIServerTracing) && !upgrade { tracingWrapper := tracing.WrapperFor(r.tracerProvider) proxyRoundTripper = tracingWrapper(proxyRoundTripper) } </ins> // If we are upgrading, then the upgrade path tries to use this request with the TLS config we provide, but it does // NOT use the proxyRoundTripper.  It's a direct dial that bypasses the proxyRoundTripper.  This means that we have to // attach the "correct" user headers to the request ahead of time.

Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted


Question: t.Error("Allocator expected to be full") } } <ins> func TestAllocateReservedDynamicBlockExhausted(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ServiceIPStaticSubrange, true)() _, storage, _, si, destroyFunc := newStorage(t) defer destroyFunc() if err := si.Create(context.TODO(), key(), validNewRangeAllocation(), nil, 0); err != nil { t.Fatalf("unexpected error: %v", err) } // allocate all addresses both on the dynamic and reserved blocks // once the dynamic block has been exhausted // the dynamic allocator will use the reserved block max := 254 for i := 0; i < max; i++ { if _, err := storage.AllocateNext(); err != nil { t.Errorf("Unexpected error trying to allocate: %v", err) } } for i := 0; i < max; i++ { ip := fmt.Sprintf("192.168.1.%d", i+1) if !storage.Has(netutils.ParseIPSloppy(ip)) { t.Errorf("IP %s expected to be allocated", ip) } } if _, err := storage.AllocateNext(); err == nil { t.Error("Allocator expected to be full") } } </ins>

In a enabled cluster, Kubernetes assigns duplicate IP addresses when the dynamic block is exhausted. I expected all clusterIPs are uniquely assigned. a enabled Kubernetes v1.24 cluster with a small service-cluster-ip-range. I confirmed on the minikube with the following command. services until the dynamic block is exhausted. see some clusterIPs are duplicated. No response <details</details<detailsminikube v1.25.2 </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
When the dynamic block is exhausted, we expect the service creation to fail with an error message like "No IP address available", right?
According to , Dynamically IP assignment will use the upper band by default, once this has been exhausted it will use the lower range. So I think using the reserved block (the lower range) is expected behavior
/sig network
upps /assign
/assign great catch, thanks


Question: gen_conversion: $(CONVERSION_FILES) if [[ -f $(META_DIR)/$(CONVERSION_GEN).todo ]]; then                 $(CONVERSION_GEN)                                                <del> -i $$(cat $(META_DIR)/$(CONVERSION_GEN).todo | paste -sd,)   </del> <ins> -i $$(cat $(META_DIR)/$(CONVERSION_GEN).todo | paste -sd, -)   </ins> -O $(CONVERSION_BASENAME);                                   fi

I don't know where the at the beginning of the target comes from, but probably from assumptions about sed version that are wrong. I had to update to gnu-sed 4.2.2 to build.
Also ootb on a Mac doesn't work with the rules specified (on el capitan):
does work
I take it back - updating sed did not appear to work. EDIT: I take that back, I had to clean after fixing paste.
most recent OSX version is UNIX certified, a POSIX requires as file argument when reading input from stdin. For , is only guaranteed to work in GNU sed, you need to insert literal newline and tab character there for portability.
Can someone with a mac send a patch? I don't have a mac...
It's update and every should end in instead of having no arguments. , Tim Hockin wrote:
(user should update sed, make needs changes to paste) , Clayton Coleman wrote:
I think we should stick to portable way for , it's too risky for relying on GNUism. If you don't mind, I will make a PR.
Sure, thanks , Cuong Manh Le wrote:
Are you sure the worked? I pull the latest from master, then run the same command above, I got: File is empty.
Make sure you are in your GOPATH in the correct spot for Kube On Jul 14, 2016, at 2:47 AM, Cuong Manh Le wrote: Are you sure the Makefile worked? I pull the latest from master, then run the same command above, I got: F0714 13:36:50. ] Error: Failed making a parser: unable to add directory "": unable to import "": expected 'package', found 'EOF' make: * [Makefile.generatedfiles:201: gendeepcopy] Error 255 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
If you get the "expected 'package', found 'EOF'" message it is usually because something somewhere failed, and left an empty .go file. make cleangenerated , Cuong Manh Le wrote:
From BSD sed man page: Maybe try that?
It's not about ERE ( enables it), it's about expanding escape sequences in RHS of search and replace .


Question: // namespaces if it's namespace scoped, so leave // APIResource.Namespaced as false is all right. apiResource := unversioned.APIResource{Name: resource.Resource} <del> // The default parameter codec used by the dynamic client cannot // encode api.ListOptions. return client.ParameterCodec(api.ParameterCodec).Resource(&apiResource, api.NamespaceAll).Watch(&options) </del> <ins> return client.ParameterCodec(dynamic.VersionedParameterEncoderWithV1Fallback). Resource(&apiResource, api.NamespaceAll). Watch(&options) </ins> }, } }

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form: