Question: volumePaths: []string{ filepath.Join("pod1", "volumes", "missing-plugin", "pvc-abcdef"), }, <del> expectedVolumesNeedReportedInUse:    []string{}, </del> expectedVolumesNeedDevicePath:       []string{}, expectedVolumesFailedReconstruction: []string{"pvc-abcdef"}, },

remove and was introduced in to fix This is to overcome the situation that volume is to DSW at first and then deleted before it is to ASW, which causes mounted volume neither in DSW nor ASW and won't execute unmount/detach: The reason the volume cannot be to ASW in time is , which is . However, there is another PR also solved the issue. What it did is to check whether ASW already has the volume before deleting it from DSW, this makes sure that volume can be detached/unmounted after it is deleted from DSW. I suggest that we can keep the second solution and remove related code to make the reconstruction process and reconcile loop cleaner?
/sig storage
cc
Should I proceed with this issue? Your guidance is appreciated.
I think we'd better wait for the feedback from
cc
Sorry for a late reply. This is how behind we are with reviews :-(. Reading Now that is on by default, it reconstructs volumes first and always puts them to ASW, it should never happen that a reconstructed volume is in DSW and not in ASW, right? So we can remove easily. I would not be so sure without , because the code is 5 years old and who knows all the implications. You could be right though, it may not be necessary there too.
/triage accepted
/assign


Question: Expect(rsList.Items[0].Spec.Template.Spec.Containers[0].Image).To(Equal(deploy.Spec.Template.Spec.Containers[0].Image)) By("Deleting the first deployment") <del> stopDeployment(c, f.Client, ns, deploy.Name) </del> <ins> stopDeploymentOverlap(c, f.Client, ns, deploy.Name, deployOverlapping.Name) </ins> // Wait for overlapping annotation cleared By("Waiting for the second deployment to clear overlapping annotation")

https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
Looking
Flakes happened when stopping overlapping deployments at clean up step.
You probably need to move the check before the deletionTimestamp check. Just make sure that observedGeneration is updated for the first case.
Because the deployments are overlapping, when deleting one deployment and determining if its RSes are deleted, there might be a race when the other deployment creates an RS
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: // certain minimum of nodes are checked for feasibility. This in turn helps // ensure a minimum level of spreading. minFeasibleNodesToFind = 100 <ins> // minFeasibleNodesPercentageToFind is the minimum percentage of nodes that // would be scored in each scheduling cycle. This is a semi-arbitrary value // to ensure that a certain minimum of nodes are checked for feasibility. // This in turn helps ensure a minimum level of spreading. minFeasibleNodesPercentageToFind = 5 </ins> ) // FailedPredicateMap declares a map[string][]algorithm.PredicateFailureReason type.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Kubernetes scheduler has that specifies what percentage of the cluster nodes should be scored in each scheduling cycle. The scheduler stops scanning more nodes once that many nodes are found feasible for running the pod under consideration. The default value of this config option is 50 (for 50%). We should let the scheduler determine this value based on the size of the cluster when this config option is not provided. I'd propose this formula for finding the percentage: . This is a linear formula that gives us 50 when the cluster size is 100 nodes and 10 for a 5000 node cluster. The scheduler already has a lower bound of 50 nodes per scheduling cycle. As a part of this change, we should also add a 5% lower bound for the formula (If the result of the formula is below 5, we should return 5). Why is this needed: To improve scheduler performance in large clusters. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature /sig scheduling
/assign /remove-help
/assign wgliang
Thanks


Question: return "", err } <ins> if svm == nil { return "", fmt.Errorf("unable to find machine reference by UUID") } </ins> var vm mo.VirtualMachine err = s.Properties(ctx, svm.Reference(), []string{"name"}, &vm) if err != nil {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Kubernetes version (use ): Environment: Cloud provider or hardware configuration: 4 VMs via ESXI 6.5 on a single physical server OS (e.g. from /etc/os-release): Ubuntu 16.04.2 Kernel (e.g. ):Install tools: kubeadm Others: docker What happened: When bootstraping a new cluster with and the cloud-provider, the cant' start because of the following panic error. Moreover, it is not possible to specify the manually in the cloud-provider config file because of the following error: What you expected to happen: My kubernetes cluster initialized on my master node with cloud-provider enabled. How to reproduce it (as minimally and precisely as possible): /etc/systemd/system/kubelet.service.d/10- /etc/kubernetes/cloud-config kubeadm init Anything else we need to know: Each VM has 2 network interfaces, one with a public IP (requirement of when doing virtualisation on a single physical server) and one with a private IP.
1 doesn't support yet, so you got this error: But I find that there is no field in your file. That's strange. Where did you configure ? Once kubelet gets the error above, it will fail to run.
I didn't put the in my example because it was just a try to get around the main issue.
As for the panic, it seems like because the below reason: The code panics : The is from : Dig into , we can find that it may return a nil : A nil will lead to the panic. cc
:) Don't know when and why the Returnval is nil. may know more about this :)
I will give a quick fix to avoid the panic.
if vmuuid is not specified in the config file, we retrieve the vm uuid at we have observed with ubuntu 16.04 incorrect uuid is present in /sys/devices/virtual/dmi/id/product_uuid with incorrect uuid, you may not be getting VM ref. Is it the issue? can you verify VM UUID using power cli. Result on Ubuntu 16.04 Result on Photon OS VM
Since you are retrieving incorrect UUID, for the FindByUUID method if no managed entities corresponding to the vm-uuid is found, null is returned. addresses this fix.

However to overcome the problem, you can specify the correct vm-uuid in the and it will read the vm reference form this UUID and it will all work fine.
Thanks for explanation! Could you please verify this on your env? Since you are retrieving incorrect UUID, for the FindByUUID method if no managed entities corresponding to the vm-uuid is found, null is returned. Since no managed entity is found in this case, should we return error here? So that can catch the error.
oh yes, thanks for the information, that was totally the issue.
We are in the process of fixing the way we fetch vmuuid. we will be parsing the serial number from which is seems tobe matching with the vmuuid we want.
I have this issue still on CentOS 7.3 with kube-controller-manager. I am running Kubernetes 1.6.4 installed using kubeadm. I have used your tip to convert productserial to the right uuid using this script: This solves the invalid memory pointer error above, but it now fails to register the master node. But that is probably a different issue not related to this issue.
We can help debugging the issue. Can you create an issue at please provide kubelet logs.
I manually enabled vsphere cloud provider after kubeadm init which solved the problem. I am happy! Thanks for your help!


Question: } func (i *internalContainerLifecycleImpl) PreStopContainer(containerID string) error { <del> if i.cpuManager != nil { return i.cpuManager.RemoveContainer(containerID) } </del> return nil } func (i *internalContainerLifecycleImpl) PostStopContainer(containerID string) error { <del> if i.cpuManager != nil { err := i.cpuManager.RemoveContainer(containerID) if err != nil { return err } } </del> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.TopologyManager) { err := i.topologyManager.RemoveContainer(containerID) if err != nil {

What happened: A container which had exclusive CPUs assigned to it (and an entry in /var/lib/kubelet/cpumanagerstate specifying those CPUs) was killed. When it came back, it was using the default cpuset, and its entry in /var/lib/kubelet/cpumanagerstate was gone. What you expected to happen: It should have been given exclusive CPUs. How to reproduce it (as minimally and precisely as possible): Create a pod/container with "guaranteed" QoS. Determine the exclusive CPUs allocated to the container. From the host, kill pid1 within the container. ("kill -9" works.) The container will restart within the existing pod, but it will be allocated cpus from the default cpuset (which will now include the CPUs that had been exclusively allocated to the container earlier). Anything else we need to know?: I some instrumentation to kubelet. What seems to be happening is this: 1) The container is killed (or dies). Kubelet then immediately creates a new container and calls PolicyStatic.Allocate() to allocate exclusive CPUs for it. This updates the entry in /var/lib/kubelet/cpumanagerstate (and the in-memory cache) which is mapped by podUID and container name, such that it contains the cpuset for the new container. 2) The InternalContainerLifecycle.PostStopContainer() code runs, which calls i.cpuManager.RemoveContainer(containerID). This uses the containerID of the exited container to look up the podUID/container-name tuple, then deletes the cpuset entry from /var/lib/kubelet/cpumanagerstate (and the in-memory cache). But the entry that was deleted was actually the one for the new container. 3) Kubelet calls the CPU manager's AddContainer() function, which calls m.state.GetCPUSetOrDefault(string(p.UID), c.Name) to get the cpuset based on the podUID and container name of the new container. The cpuset entry was erroneously deleted, so this returns the default cpuset, and the container which should have gotten exclusive CPUs ends up running on the default cpuset. An solution suggested by is to delay calling cpuManager.RemoveContainer() until the whole pod is deleted. This would preserve the CPU allocations.

An alternate solution also suggested by is to avoid calling cpuManager.RemoveContainer() from the container lifecycle code, but instead call removeStaleState() in the CpuManager.Allocate() function. Environment: Kubernetes version (use ): 1.18.1 Cloud provider or hardware configuration: x86, SMP OS (e.g: ): CentOS 7 Kernel (e.g. ): 3.10.0-1062.1.2.rt56.1025.el7
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Considering the following change based on discussion over Slack with
Sorry that bother you via this close issue, but I was tracing back to this issue via the PR. As per your statement above, if I understand correctly, this problem only happened when the container inside Pod got killed. but an simple Pod restarted (e.g. delete/re-create via kubectl delete pod command) should not trigger the problem, right? Or any container(inside Pod) or Pod restart with guaranteed Qos will trigger this issue?


Question: permitPlugin.waitAndRejectPermit = test.waitReject permitPlugin.waitAndAllowPermit = test.waitAllow <del> // Create two pods. waitingPod, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "waiting-pod", Namespace: testCtx.NS.Name})) </del> <ins> // Create two pods. First pod to enter Permit() will wait and a second one will either // reject or allow first one. podA, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "pod-a", Namespace: testCtx.NS.Name})) </ins> if err != nil { <del> t.Errorf("Error while creating the waiting pod: %v", err) </del> <ins> t.Errorf("Error while creating the first pod: %v", err) </ins> } <del> signallingPod, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "signalling-pod", Namespace: testCtx.NS.Name})) </del> <ins> podB, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "pod-b", Namespace: testCtx.NS.Name})) </ins> if err != nil { <del> t.Errorf("Error while creating the signalling pod: %v", err) </del> <ins> t.Errorf("Error while creating the second pod: %v", err) </ins> } if test.waitReject { <del> if err = waitForPodUnschedulable(testCtx.ClientSet, waitingPod); err != nil { t.Errorf("test #%v: Didn't expect the waiting pod to be scheduled. error: %v", i, err) </del> <ins> if err = waitForPodUnschedulable(testCtx.ClientSet, podA); err != nil { t.Errorf("test #%v: Didn't expect the first pod to be scheduled. error: %v", i, err) </ins> } <del> if err = waitForPodUnschedulable(testCtx.ClientSet, signallingPod); err != nil { t.Errorf("test #%v: Didn't expect the signalling pod to be scheduled. error: %v", i, err) </del> <ins> if err = waitForPodUnschedulable(testCtx.ClientSet, podB); err != nil { t.Errorf("test #%v: Didn't expect the second pod to be scheduled. error: %v", i, err) } if !((permitPlugin.waitingPod == podA.Name && permitPlugin.rejectingPod == podB.Name) || (permitPlugin.waitingPod == podB.Name && permitPlugin.rejectingPod == podA.Name)) { t.Errorf("test #%v: Expect one pod to wait and another pod to reject instead %s waited and %s rejected.", i, permitPlugin.waitingPod, permitPlugin.rejectingPod) </ins> } } else { <del> if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, waitingPod); err != nil { t.Errorf("test #%v: Expected the waiting pod to be scheduled. error: %v", i, err) </del> <ins> if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, podA); err != nil { t.Errorf("test #%v: Expected the first pod to be scheduled. error: %v", i, err) } if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, podB); err != nil { t.Errorf("test #%v: Expected the second pod to be scheduled. error: %v", i, err) </ins> } <del> if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, signallingPod); err != nil { t.Errorf("test #%v: Expected the signalling pod to be scheduled. error: %v", i, err) </del> <ins> if !((permitPlugin.waitingPod == podA.Name && permitPlugin.allowingPod == podB.Name) || (permitPlugin.waitingPod == podB.Name && permitPlugin.allowingPod == podA.Name)) { t.Errorf("test #%v: Expect one pod to wait and another pod to allow instead %s waited and %s allowed.", i, permitPlugin.waitingPod, permitPlugin.allowingPod) </ins> } }

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose?

I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.


Question: expectFatal:  false, expectDelete: false, }, <ins> { description:                "fail to list pods", node:                       node, expected:                   cordonedNode, pods:                       []corev1.Pod{rsPod}, replicaSets:                []appsv1.ReplicaSet{rs}, args:                       []string{"node"}, expectFatal:                true, expectDelete:               true, failUponEvictionOrDeletion: true, }, </ins> } testEviction := false

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl drain crashed with panic when apiserver connection disappeared while there were pods that couldn't be evicted/deleted (sorry, the problem is a bit hard to reproduce and the log was lost) What you expected to happen: kubectl drain should exit with a proper error message How to reproduce it (as minimally and precisely as possible): Try kubectl drain multiple times, interrupting the connection during eviction/deletion Anything else we need to know?: The here can be in case if returns an error, and will panic upon accessing : This causes the c Environment: Kubernetes version (use ): The problem is present as of but is also present in earlier versions (I have a fix for this bug)
/priority important-longterm /sig cli


Question: // Returns an interactive clientConfig if the password flag is enabled, // or a non-interactive clientConfig otherwise. func (f *ConfigFlags) ToRawKubeConfigLoader() clientcmd.ClientConfig { <ins> if f.usePersistentConfig { return f.toRawKubePersistentConfigLoader() } return f.toRawKubeConfigLoader() } func (f *ConfigFlags) toRawKubeConfigLoader() clientcmd.ClientConfig { </ins> loadingRules := clientcmd.NewDefaultClientConfigLoadingRules() // use the standard defaults for this client command // DEPRECATED: remove and replace with something more accurate

/kind bug What happened: takes ~11 seconds when there are thousands of CRDs and is around 28KiB. logs the following message once per resource: Removing unnecessary entries from reduces the runtime by around ten seconds. What you expected to happen: only parses once, and therefore doesn't run unnecessarily slowly. How to reproduce it (as minimally and precisely as possible): shows the config being loaded multiple times (assuming you have multiple pods in the current cluster). Alternatively, add 10-15 clusters to and create ~3000 of a cheap resource (eg custom resources or empty config maps). Run on this resource and observe that it is unnecessarily slow. Anything else we need to know?: I've created a CPU profile using the latest CI build of kubectl, which shows ~12 seconds being spent in . Environment: Kubernetes version (use ): v1.12.1 (also Cloud provider or hardware configuration: GKE or minikube --vm-driver=none OS (e.g. from /etc/os-release): Debian Buster-like Kernel (e.g. ): 4.17.0
/sig cli
/assign
Same problem here. I have less resources but more contexts in my kubeconfig file. With 90 pods and ~300 contexts in the kubeconfig it also takes ~10 seconds for a simple kubectl get pods Note: I tried it with kubectl: 1.11.0, 1.12. With kubectl 1.10.5 the kubeconfig was loaded only once.
I'm unable to reproduce this as of the latest master. My kubeconfig is only loaded 4 times, no matter how many resources are loaded.
When I try a kubectl v1.13.0-alpha.3 against unfortunately a 1.9.11 Kubernetes Cluster I still have the problem: `
Maybe the bug is Linux-only? Presumably the kubeconfig location is different on different platforms, so maybe it has a different code path. I can repro on latest master:
I filed to fixed this. You can try to help validate it. Thanks.
Works for me, thanks!


Question: NodeName:                 node.Name, } <del> // Disable volume lifecycle checks due to issue #103651 for the one // test that it breaks. // TODO: enable this check once issue is resolved for csi-host-path driver // (https://github.com/kubernetes/kubernetes/pull/104858). if regexp.MustCompile("should unmount if pod is.*deleted while kubelet is down").MatchString(ginkgo.CurrentGinkgoTestDescription().FullTestText) { o.DriverContainerArguments = append(o.DriverContainerArguments, "--check-volume-lifecycle=false") } </del> cleanup, err := utils.CreateFromManifests(config.Framework, driverNamespace, func(item interface{}) error { if err := utils.PatchCSIDeployment(config.Framework, o, item); err != nil { return err

When a pod is marked as deleted while kubelet is down / being restarted, newly started kubelet does not clean up CSI filesystem volumes of the pod. Newly started kubelet tries to reconstruct the volume using CSI's function. This part looks working, CSI volume plugin loads its json file. But then VolumeManager checks if the volume is still mounted in directory. There are two issues: does not require volumes to be presented as mounts. They can be just directories with files on them. This will be case of the most of in-line volumes. Kubelet checks the right directory given by
/sig storage related to cc
Maybe a stupid idea... Reconciler sync has found a directory in . Reconciler has no way how to check that CSI volume (or any other volume) is really mounted / published there. Depending on DSW: The volume is in DSW -from comment here it seems that the SetUp (=NodePublish) will be called on the volume: The volume is not in DSW and it will be to ASW, so it's unmounted / unpublished in the next sync: In both cases, we can expect that the volume plugin / CSI driver is idempotent and SetUp() won't do anything if the volume is already set up / TearDown() will not do anything if the volume has been torn down already. Are our volume plugins really idempotent? IMO they are. We perhaps don't need to check for presence of mount point anywhere, just presence of directory should be enough to reconstruct the volume and set it up / tear it down. Adding to the loop.
Filed , PTAL
I'm curious how the reconstruction e2e tests passed
https://k8s-*force
They start kubelet with a pod already deleted. If the pod still exists, volume reconstruction can see it in DSW and does nothing when fails: [Reconstruction fails because it checks for a mount in a wrong directory.] In this case, kubelet waits for the volume to appear in VolumesInUse before calling . The pod is deleted before that and kubelet then just deletes DSW entry and the volume is never cleaned up.
They pass because their container does not handle SIGTERM and it takes 30 seconds to kill them. Kubelet has enough time to the volume during reconciliation.

It turns out that I tested with broken version of and reconstruction is broken only for ephemeral volumes that don't mount.
So the conclusion is that we only need to fix for ephemeral volumes? Thanks!
Yes, sorry again about the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle rotten
/lifecycle frozen
From this issue surfaced through e2e tests using the hostPath CSI driver. The cause is (which is used in for reconstruction) - it doesn't play well with volumes that bind mount from a directory instead of a device.
To work around this bug here, volume life cycle checking was disabled for all tests, not just the subpath test: This check is useful and should be enabled again.
Hey everyone! I arrived at this conversation by investigating on issue (seems to me like a duplicate of ). I'm wondering if there's something I can help with? I'm still a bit new to the codebase, though. I was able to reproduce the problem by using the following ginkgo focus command in the e2e framework: And do see that if we manually add the --check-volume-lifecycle=false to the statefulset/csi-hostpathplugin tests will pass correctly. I also see that during the process, something in the cleanup function doesn't correctly delete the pv's after the tests are done, so they remain alive if we execute a kubectl get pv command.
I think I found a bit of more info on the issue probably could be of some help. I think that the volume is not being correclty marked as unmounted inside in GenerateUnmountVolumeFunc. I'm not sure why but I'll continue to investigate on it. I think this error is coming from DeletePodFromVolume in
I a couple of extra debug lines to that lines in the file, and can see that when DeletePodFromVolume tries to see if the volume exists, it doesn't find it in the asw.attachedVolumes struct, like so: Probably we should search for the name of the attached volume differently for ephemeral volumes?
/assign
There are two problems: 1) The IsLikelyNotMountPoint() call that , which is known not to work for bind mounts. The mountpoint never gets found, so volume reconstruction fails with "... is not mounted", the PVC never never gets to the list of attached volumes, and that's why we end up hitting this error: 2) CSI inline volumes hit a different error (in addition to the first problem). The call path looks like this (starting in ): Inline volumes do not have a PV spec, so it fails . We should be calling instead.

There is an ongoing that changes to utilize syscall to detect mount point by using . By using , bind mount will be properly detected fast, but it requires kernel version is v5.6 or later. So, we would be able to also utilize in . Then, the issue of bind mount can be resolved at least for kernel v5.6+, and we will be able to focus on how we solve this issue for old kernels.


Question: assert.Equal(t, owns, c.expected, "TestCase[%d]: %s", i, c.desc) } } <ins> func TestGetServiceTags(t *testing.T) { tests := []struct { desc     string service  *v1.Service expected []string }{ { desc:     "nil should be returned when service is nil", service:  nil, expected: nil, }, { desc:     "nil should be returned when service has no annotations", service:  &v1.Service{}, expected: nil, }, { desc: "single tag should be returned when service has set one annotations", service: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationAllowedServiceTag: "tag1", }, }, }, expected: []string{"tag1"}, }, { desc: "multiple tags should be returned when service has set multi-annotations", service: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationAllowedServiceTag: "tag1, tag2", }, }, }, expected: []string{"tag1", "tag2"}, }, { desc: "correct tags should be returned when comma or spaces are included in the annotations", service: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationAllowedServiceTag: ", tag1, ", }, }, }, expected: []string{"tag1"}, }, } for i, c := range tests { tags := getServiceTags(c.service) assert.Equal(t, tags, c.expected, "TestCase[%d]: %s", i, c.desc) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider has a limited support for service tags by annotations: But according to , there are actually much more service tags than this list. Hence, errors would happen if the service tag is not included in the above list. What you expected to happen: Azure cloud provider should support all service tags from Azure. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig azure /priority important-soon
/assign


Question: import ( "fmt" <del> "io" </del> <ins> "io/ioutil" </ins> "net" <ins> "net/http" "net/http/httptest" "net/url" </ins> "strconv" "testing" "time"

https://travis-
Looking.
Another instance, or a different thing? https://travis-


Question: // A maxItems value of -1 means no limit. func NewTimedStore(age time.Duration, maxItems int) *TimedStore { return &TimedStore{ <del> buffer:   make([]timedStoreData, 0), </del> <ins> buffer:   make(timedStoreDataSlice, 0), </ins> age:      age, maxItems: maxItems, }

I have some reports of a Node reporting a a resource capacity for cpu as a negative value. Any idea why cpu could report as a negative value? Is this an expected normal outcome? In its current state, it prevents any pods from being scheduled to that Node. Tips on how to debug further are appreciated. - any ideas?
Can you check machineInfo on Cadvisor or kubelet on the node?
a.k.a:
It is a bug in cAdvisor side. I am assigning it to you :-)
here is contents of /proc/cpuinfo Pertinent pieces of a log:
Machine is AWS instance if that matters....
We build the topology from system/cpu info. I think we can return partial topology information and ignore caches if its not present. This isn't a recent change though. Is it happening on just a single machine? I'll file a cAdvisor issue.
Can you link the referenced issue once created?
Re-assigning to since he's working on google/cadvisor
Could be this was caused by getting a Node on EC2 with old hypervisor, and I am the first victim...
That sounds likely. Most of the cache information is there in /proc/cpuinfo. So looks like the sysfs bits are disabled.
Can you try this cadvisor image on your machine and check /api/v1.3/machine output: sudo docker run  --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw  --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080  --detach=true  --name=cadvisor  jnagal/cadvisor:canary
hmm
ah, selinux :( You'll need to run it with --privileged, I think. who's been dealing with it.
No luck with --privileged either on this box...
This has to do with SELinux, I think we had to disable it for this to work (some of our mounts are being denied). We're working on a better solution... This is not a problem in Kubernetes since we don't run Kubelet/cAdvisor in a Docker container.
ok, unfortunately, i have managed to kill docker on the box in question, and i cannot get the daemon to restart right now so I am not sure when that box will be available to debug against again.

I can provide an environment with a similar issue if required.
The fix is checked in to head. Can you build the binary and try?


Question: var outputObj runtime.Object obj := info.Object <del> if o.dryRunStrategy == cmdutil.DryRunClient || o.local { </del> <ins> if o.dryRunStrategy == cmdutil.DryRunClient || o.local || o.list { </ins> if err := o.updateAnnotations(obj); err != nil { return err }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : add --list option to kubectl annotate command Why is this needed: like kubectl label commandshow annotations quickly
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign


Question: } }() } <del> // unhold all the calls with the same namespace handler.GetLimitRanges(attributes) calls, that have to be aggregated unhold <- struct{}{} go func() { unhold <- struct{}{} }() </del> // and here we wait for all the goroutines wg.Wait() // since all the calls with the same namespace will be holded, they must be catched on the singleflight group, // There are two different sets of namespace calls // hence only 2 <del> if count != 2 { t.Errorf("Expected 1 limit range, got %d", count) </del> <ins> if testCount != 1 { t.Errorf("Expected 1 limit range call, got %d", testCount) } if test1Count != 1 { t.Errorf("Expected 1 limit range call, got %d", test1Count) </ins> } // invalidate the cache handler.liveLookupCache.Remove(attributes.GetNamespace()) <del> go func() { // unhold it is blocking until GetLimitRanges is executed unhold <- struct{}{} }() </del> _, err = handler.GetLimitRanges(attributes) if err != nil { t.Errorf("unexpected error: %v", err) } <del> close(unhold) if count != 3 { t.Errorf("Expected 2 limit range, got %d", count) </del> <ins> if testCount != 2 { t.Errorf("Expected 2 limit range call, got %d", testCount) } if test1Count != 1 { t.Errorf("Expected 1 limit range call, got %d", test1Count) </ins> } }

pull-kubernetes-unit /cc /sig scalability
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In local tests, he always passes, so I'll see if the problem is caused by the environment
If the lru cache exceeds 30 seconds, the cache will be invalidated, which will lead to unit test errors, and the problem will run normally again after retesting.
After I manually reduce an unhold <- struct{}{}, the same problem occurs, so it seems that the problem is caused by the cache being invalidated for more than 30 seconds. If the test doesn't complete in 30 seconds, then this is probably a machine performance issue
stress will hang after about a minute in my local test.
/cc
The issue is being fixed


Question: gotNames.Insert(rs.Name) } <del> assert.Equal(t, 0, expectedNames.Difference(gotNames).Len(), "expected %v, got %v", expectedNames.List(), gotNames.List()) assert.Equal(t, 0, gotNames.Difference(expectedNames).Len(), "expected %v, got %v", expectedNames.List(), gotNames.List()) </del> <ins> if diff := cmp.Diff(expectedNames.List(), gotNames.List()); diff != "" { t.Errorf("Active replica set names (-want,+got):n%s", diff) } </ins> } func TestComputeHash(t *testing.T) {

During implementation of KEP 3393 (PodReplacementPolicy for Jobs) we found that the unit tests of do not follow common patterns established in k/k. 1) The tests are not written using table driven design (). 2) The tests use rather than . This probably should be done as two separate PRs but wanted to highlight some areas that could use some cleanup. Not using means that we don't see diffs in the unit tests that makes this difficult. And not using table driven tests makes it difficult to add new test cases as it usually is brought up during review that we should use a table-driven test but the entire file does not obey this format.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig apps /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on it /assign
I'd also like to make some contributions. /assign
Looks like this need two PRs. tests are not written using table driven design (). tests use Differences rather than Cmp.Diff. I think I can work on the 2nd one and leave the 1st one to you.

Sounds good?
By looking through the , I found the main function to comapre the results have already used cmp.Diff. Can you elaborate more on which part you expect to use cmp.Diff? func verifyDeletedAndPatchedPods(t testing.T, client fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client) if diff := cmp.Diff(wantDeletedPodNames, deletedPodNames); diff != "" { t.Errorf("Deleted pod names (-want,+got):n%s", diff) } patchedPodNames := getPatchedPodNames(client) if diff := cmp.Diff(wantPatchedPodNames, patchedPodNames); diff != "" { t.Errorf("Patched pod names (-want,+got):n%s", diff) } }
I am really sorry about this. Too much multitasking!! I meant not
Hey ! Sorry, I didn't see this before. No problem at all, but consider that one change depends on the other, and since my task implies a big refactor, there might be conflicts when merging your PR... That's why I see two possible scenarios happening: Your PR gets merged before I raise mine: Then fine, I'll rebase your changes and adapt my code Mine is ready before yours is merged: In this case, I propose you commit your changes to my PR. Both of us will appear as coauthors of the change Getting a PR merged in the Kubernetes project takes quite some time (trust me, I talk by experience and can confirm ) WDYT?
Let's see how it goes. We can always rebase the code change to adapt new.
btw, did you already begin to work on task 1? If not, I may just go ahead and work on it right away since I have already dig into it.
yes, I already started And If you ask me, better to keep it this way so we both have an opportunity to have a contribution
cool. Sounds good.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi if this issue needs to be worked on any further, please assign me for the same.
has a PR up that will solve the second part of this issue.


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework"

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // All volumes bound or matching PVs found for all unbound PVCs <del> glog.V(5).Info("All PVCs found matches for pod %v/%v, node %q", pod.Namespace, pod.Name, node.Name) </del> <ins> glog.V(5).Infof("All PVCs found matches for pod %v/%v, node %q", pod.Namespace, pod.Name, node.Name) </ins> return true, nil, nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: Volume got bound to PV on node "kubernetes-minion-group-82c8" When the pod went back through scheduling, the predicate returned success for other nodes even though we should have updated the PV cache to point to the PVC: I think in , instead of continuing if the node affinity doesn't match, it should fail if the PV is prebound.
This causes two issues: On the second pass, the scheduler may choose a different node. Then in Assume/Bind volumes, we'll also prebind a second set of PVs. If the scheduler chose a different node on the second pass, and the PVCs became fully bound in between the predicates and assume volumes, then assume volumes will return success and the pod will get scheduled to the wrong node.


Question: }, } <del> expectedCandidates := []*kubecontainer.ContainerStatus{pod.ContainerStatuses[2], pod.ContainerStatuses[1]} candidates := newPodContainerDeletor(&containertest.FakeRuntime{}, 1).getContainersToDeleteInPod("2", &pod) if !reflect.DeepEqual(candidates, expectedCandidates) { t.Errorf("expected %v got %v", expectedCandidates, candidates) </del> <ins> testCases := []struct { containersToKeep           int expectedContainersToDelete []*kubecontainer.ContainerStatus }{ { 0, []*kubecontainer.ContainerStatus{pod.ContainerStatuses[3], pod.ContainerStatuses[2], pod.ContainerStatuses[1]}, }, { 1, []*kubecontainer.ContainerStatus{pod.ContainerStatuses[2], pod.ContainerStatuses[1]}, }, { 2, []*kubecontainer.ContainerStatus{pod.ContainerStatuses[1]}, }, } for _, test := range testCases { candidates := getContainersToDeleteInPod("4", &pod, test.containersToKeep) if !reflect.DeepEqual(getContainersToDeleteInPod("4", &pod, test.containersToKeep), test.expectedContainersToDelete) { t.Errorf("expected %v got %v", test.expectedContainersToDelete, candidates) } </ins> } }

Kubelet evicts pods whenever there is memory or disk pressure. It marks the pods as to evict them. To help with reducing disk pressure, kubelet should delete all containers belonging to pods right away. This will result in loss of logs. In the future, we can copy out logs from dead containers before deleting them. Logs can be rotated separately from dead containers. IMO, this feature is required for disk usage based evictions to work reliably in kubelet.


Question: var responseTimes, creationTimes []time.Duration var ingController *IngressController <del> f := framework.Framework{BaseName: "glbc"} </del> <ins> f := framework.NewDefaultFramework("glbc") </ins> BeforeEach(func() { // This test requires a GCE/GKE only cluster-addon

Sometime yesterday the test started panicking because the framework is nil. No changes were made to the test, did we change framework setup? This is the stack
Aarg now it looks like we inverted the order of running framework.AfterEach and custom AfterEach, so I re-invoke framework.AfterEach in my AfterEach (we used to need this) and it crashes. I'm just going to usher in as it doesn't invoke framework.AfterEach. For the record this will keep failing on 1.3 till the is cherrypicked, but the diff between 1.3 and HEAD is insignificant at this point (and remain insignificant till the cherrypick happens)
Now gce-1.3 passes but will fail because went in without They should pass when is cherrypicked to 1.3 ( for context)


Question: } dest.Paths.Paths[k] = v } <ins> return nil </ins> }

Related to Seeing the following errors in kube-apiserver logs when CRDs are applied: No errors in logs Apply any CRD manifest No response <details</details<details</details<details</details<detailsRancher RKE2 </details<detailscontainerd 1.6.4 </details<detailscanal </details
/sig api-machinery
/assign
Did this cause any specific trouble to your cluster I'm curious what the impact/severity of this is.
Everything seems to work fine, but there's a bunch of log spew every time a CRD manifest is loaded. In K3s we mix all the control-plane component logs together in a single pair of stdout/stderr streams so it's quite noisy, compared to previous releases. The fact that these come out at WARN level with a scary prefix makes them look pretty serious, and I expect we will get a fair bit of users asking about it once we release.
/triage accepted


Question: go_rules_dependencies() go_register_toolchains( <del> go_version = "1.11.2", </del> <ins> go_version = "1.11.3", </ins> ) docker_repositories()

The go team has released 1.11.3 with three security fixes. From their e-mail announcement: We should kick off upgrades of both the kubernetes build system, and our CI, as soon as possible. Do you have an ETA for the rules_go bump that will contain 1.11.3 toolchain support? cc:
I'm preparing releases that use Go 1.11.3 by default. Those will be on the rules_go 0.14, 0.15, and 0.16 branches. I expect to tag them tonight as soon as they get through CI.
Fantastic! Thank you
rules_go 0.14.7, 0.15.9, and 0.16.4 are now tagged and ready to go.
related commits and issue:
Reopening to track other branches, as well as the CI bump
go1.11.3 and go1.10.6 introduced a regression, and so go1.11.4 and go1.10.7 were released Friday afternoon with a fix. We'll need to bump everything again on master, and should hold off on cherry-picks for now.
This looks fixed by Is there anything else that needs to be done as well?
Still need to merge for k8s 1.13.
Closing as all branches are now up to date.


Question: readonly UUID_MNT_PREFIX="/mnt/disks/by-uuid/google-local-ssds" readonly UUID_BLOCK_PREFIX="/dev/disk/by-uuid/google-local-ssds" <ins> readonly COREDNS_AUTOSCALER="Deployment/coredns" readonly KUBEDNS_AUTOSCALER="Deployment/kube-dns" </ins> # Use --retry-connrefused opt only if it's supported by curl. CURL_RETRY_CONNREFUSED=""

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).


Question: } func validateProxyMode(mode kubeproxyconfig.ProxyMode, fldPath *field.Path) field.ErrorList { <ins> if runtime.GOOS == "windows" { return validateProxyModeWindows(mode, fldPath) } return validateProxyModeLinux(mode, fldPath) } func validateProxyModeLinux(mode kubeproxyconfig.ProxyMode, fldPath *field.Path) field.ErrorList { </ins> allErrs := field.ErrorList{} switch mode { case kubeproxyconfig.ProxyModeUserspace: case kubeproxyconfig.ProxyModeIPTables:

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When running , and passing the flag which was introduced in , the following error occurs: What you expected to happen: The proxy should work as it used to, supporting this mode. How to reproduce it (as minimally and precisely as possible): ./kube- --proxy-mode=kernelspace Anything else we need to know?: This was introduced after the validation was refactored in which regressed validation for . IPVS validation (and thus support) was later , but we need another mode for Windows. Environment: Kubernetes version (use ): v1.10.0-alpha.0.684+ Cloud provider or hardware configuration: n/a OS (e.g. from ): Windows Server Core 1709 Kernel (e.g. ): Install tools: Others:
/sig windows /sig network
Tentatively adding milestone so we can track getting consensus on release-blocking status.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: return client.TokenReviews(), nil } <ins> type ignorableError struct{ error } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When I start the core API server with , it populates the ConfigMap in namespace with the requestheader CA. This allows the extension API server to dynamically discover the CA and use it for validating the client certificate of aggregator layer. See for documentation However, if the core API server doesn't use certificate based authentication for it's clients i.e. flag isn't set in core API server then the extension API server fails to start with error message I believe this is coming from bootstrap code in extension API server . What you expected to happen: Extension API server should be able to start and not require that both and be present in the ConfigMap. This allows Kubernetes clusters which don't use certificate based authentication for core API server to be able to dynamically discover the requestheader-ca instead of having to mount it as a secret. How to reproduce it (as minimally and precisely as possible): Create a Kubernetes cluster which uses non certificate based authentication for core API server like Webhook based authentication. Example: You can launch an Amazon EKS cluster which uses Webhook based authentication following steps outlined . Setup aggregator layer by adding flags mentioned . CA certificate and client certificate for aggregator layer can be generated using and . Deploy metrics server to cluster following steps in Deployment section of README Get logs of metrics server pod and it should be failing with error message above Anything else we need to know?: It's possible there is a reason which is not evident to me as to why extension API server would require both client-ca-file and requestheader-client-ca-file to be present in the ConfigMap. Environment: Kubernetes version (use ): 1.10.3 Cloud provider or hardware configuration: Amazon EKS OS (e.g. from /etc/os-release): Kubernetes control plane and nodes = Amazon Linux 2 (2017.12) LTS Release Candidate (4.14.42).
/sig api-machinery /cc
since it looks like the at-mention didn't take
/sig auth /assign
Thanks Jordan. This is something we are interested in contributing to and ask for being cherrypicked to patch releases of 1.10.x and 1.11.x. That conversation can wait once we have the PR out.

Quick update: I'm working on sending a PR for this issue by this week
- Now that this is merged, can we open up the possibility of cherry picking to 1.10.x and 1.11.X please?
sure, opened , ,


Question: // CommonAccessor returns a Common interface for the provided object or an error if the object does // not provide List. <del> // TODO: return bool instead of error </del> func CommonAccessor(obj interface{}) (metav1.Common, error) { switch t := obj.(type) { case List:

Anything else we need to know?: I notice there are TODOs in apimachinerypkgapimeta,return bool instead of error func CommonAccessor, ListAccessor, Accessor. If is it necessary to do,I push a PR for it. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: for shortName, imageConfig := range externalImageConfig.Images { var images []string isRegex, name := false, shortName <del> if imageConfig.ImageRegex != "" && imageConfig.Image == "" { </del> <ins> if (imageConfig.ImageRegex != "" || imageConfig.ImageFamily != "") && imageConfig.Image == "" { </ins> isRegex = true <del> images, err = getGCEImages(imageConfig.ImageRegex, imageConfig.Project, imageConfig.PreviousImages) </del> <ins> images, err = getGCEImages(imageConfig.ImageRegex, imageConfig.ImageFamily, imageConfig.Project, imageConfig.PreviousImages) </ins> if err != nil { <del> klog.Fatalf("Could not retrieve list of images based on image prefix %q: %v", imageConfig.ImageRegex, err) </del> <ins> klog.Fatalf("Could not retrieve list of images based on image prefix %q and family %q: %v", imageConfig.ImageRegex, imageConfig.ImageFamily, err) </ins> } } else { images = []string{imageConfig.Image}

We should support image family in node e2e image config. It makes image validation much easier on GCE.
Cool idea :) Adding a link to for anyone like me who hadn't heard of them before :)


Question: "crypto/x509" "fmt" "io/ioutil" <ins> "strings" </ins> "github.com/Azure/go-autorest/autorest/adal" "github.com/Azure/go-autorest/autorest/azure"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need to enhance Azure cloud provider code to support both AAD and ADFS authentication. Why is this needed: On Azure Stack customer can have two type of deployment .
/sig azure
/assign


Question: k8s.io/mount-utils v0.0.0 k8s.io/pod-security-admission v0.0.0 k8s.io/sample-apiserver v0.0.0 <del> k8s.io/system-validators v1.5.0 </del> <ins> k8s.io/system-validators v1.6.0 </ins> k8s.io/utils v0.0.0-20210930125809-cb0fa318a74b sigs.k8s.io/structured-merge-diff/v4 v4.2.0 sigs.k8s.io/yaml v1.2.0

every k8s release, if changes are to we release a new version of the library and vendor it in this repository. TODO: [ ] wait for changes to accumulate in [ ] add new release in before the k8s 1.23 release [ ] vendor the new release in k/k (example PR: )
/sig cluster-lifecycle node /milestone v1.23 /assign
/triage accepted
/cc


Question: var ( // TODO: Deprecate gitMajor and gitMinor, use only gitVersion instead. gitMajor     string = "0"              // major version, always numeric <del> gitMinor     string = "8.2+"           // minor version, numeric possibly followed by "+" gitVersion   string = "v0.8.2-dev"     // version from git, output of $(git describe) </del> <ins> gitMinor     string = "8.3"            // minor version, numeric possibly followed by "+" gitVersion   string = "v0.8.3"         // version from git, output of $(git describe) </ins> gitCommit    string = ""               // sha1 from git, output of $(git rev-parse HEAD) gitTreeState string = "not a git tree" // state of git tree, either "clean" or "dirty" )

The web and instructions for [Installing and Setting Up kubectl] need a slight tuneup. The homebrew instructions should reflect a few different options for Bash on macOS. bash-completion is for Bash 3.2 (macOS default), and bash-completion is for Bash 4.1+ (upgraded GNU Bash) I lost an hour of my life to this; I hope this helps others. What happened: By following the homebrew install instructions on this page, then entering and tabbing, this is the output: What you expected to happen: By following the same path as above this should be the output: Anything else we need to know: Currently the instructions assume the default Bash on a vanilla macOS install; while I understand this being most-likely scenario, the default version of Bash is shamefully old. After reviewing to options in homebrew it appears there are 2: -completion is for Bash 3.2 and -completion is for Bash 4.1+ For those of us that cannot live without all the options, this is how we do: Install: Configure Then the needs to be updated to reflect this changes: TEST a new tab (the is sourced-in) and tab twice We then get the expected result. The requested details follow: Kubernetes version (use ): Environment: personal/dev Cloud provider or hardware configuration: NA OS (e.g. from /etc/os-release): macOS Sierra Kernel (e.g. ): Install tools: homebrew [Installing and Setting Up kubectl]
So were you running bash 3.2 or bash 4 prior to this? It looks like the instructions are for bash-completion which should be for the bash installed by default in macOS. /assign
I always install the latest Bash, currently 4.4.12(1)-release. Apple's is too old and I'm not going live in their cave; most people won't - that's why homebrew is a huge project. Correct, the instructions are for the stock version of Bash (presently 3.2.57(1)-release). My suggestion is to include instructions for both versions. Mostly to keep others from wasting the same time. I've already done the testing so folding in the new details should be pretty straight-forward.
I've opened to update the kubernetes documentation, and to potentially update how homebrew works to make this clearer.
Docs have been updated: /close


Question: minQuotaPeriod = 1000 ) <del> // milliCPUToShares converts milliCPU to CPU shares func milliCPUToShares(milliCPU int64) int64 { if milliCPU == 0 { // Return 2 here to really match kernel default for zero milliCPU. return minShares } // Conceptually (milliCPU / milliCPUToCPU) * sharesPerCPU, but factored to improve rounding. shares := (milliCPU * sharesPerCPU) / milliCPUToCPU if shares < minShares { return minShares } return shares } </del> // milliCPUToQuota converts milliCPU to CFS quota and period values func milliCPUToQuota(milliCPU int64, period int64) (quota int64) { // CFS quota is measured in two values:

Deployment of a POD onto a machine with 488 cores fails if the POD gets deployed with QoS Guaranteed or Burstable when cpu request is larger than 256. Kubelet Error Trace: RunContainerError: failed to start container "4ad8bf2f988c9c387d8136ac664caf4af80648ac436e92dac6ecd1f9aece9144": Error response from daemon: OCI runtime create failed: starting container process caused " container init caused " setting cgroup config for procHooks process caused "The maximum allowed cpu-shares is """: unknown Normal start of the container with the right number of cpu-shares set yo the cgroup the POD runs in. Install kubernetes on a machine with more than 300 cores and deploy a POD with QoS Guaranteed or Burstable where the cpu-request value is set to more than 256 cpus. Deployment without CPU request and limits works. Deployments with cpu-request <= 256 works as well. I found this change: Unfortunately the changed code does not get used in all cases - is not used in my case. Kubernetes version (use ): Kubernetes v1.19.8 Cloud provider or hardware configuration: AWS / EC2 u- (6 TB memory / 488 vCPUS) OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp1" VARIANT_ID="chost" Kernel (e.g. ): Linux ip-10-250-0- 4.12.14-197.72-default Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Hi thanks for raising the issue. Can you share the pod spec(maybe the part that includes CPU shares) that caused the issue?

Also which container runtime are you using? And its version?
I think might be related to the error message you're seeing. That repo as you can see is archived so I think you should try upgrading your container runtime. Also in my understanding, CPU share is a relative value that tells kernel scheduler how to schedule the cpu time slice to each task. In my understanding we should expect a large cpu quota here, rather than a large CPU share. Just thinking loud.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. Kernel code where max limits are set: /triage accepted /remove-lifecycle stale It seems we are only clamping for stuff under , I don't know that this is being applied to all pods. /cc I think there's two things we should do: this limit more clearly the clamping to ensure we don't run into this error /kind documentation /help Help requested with improving the docs.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. /remove-lifecycle stale Unfortunately the changed code does not get used in all cases - is not used in my case. Update the clamping to ensure we don't run into this error /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the issue is in the Kubelet. The kernel uses the range for , but the kubelet can generate and request values outside of the valid range. When using cgroupfs, the kernel automatically clamps the value.

Instead systemd has an additional check and refuses to use an invalid value (even if the kernel would accept and clamp it). The clamping is just to make sure the systemd driver works in the same way as cgroupfs, but now any value set above the valid range will get assigned the same number of . would you mind opening a PR to add the clamping also for your case?
I would like to take this one by opening a PR for the clamping issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
have you had a chance to look at this issue?
I just created the PR but not sure if I it to the right place. Could you take a look?
/remove-lifecycle stale
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: <del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> package(default_visibility = ["//visibility:public"]) load( "@io_bazel_rules_go//go:def.bzl", "go_library", "go_test", ) </ins> go_library( name = "go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are no unit tests yet for pkg , we should add unit tests. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign


Question: } } <ins> // Drop the packets in INVALID state, which would potentially cause // unexpected connection reset. // https://github.com/kubernetes/kubernetes/issues/74839 writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-m", "conntrack", "--ctstate", "INVALID", "-j", "DROP", ) </ins> // If the masqueradeMark has been added then we want to forward that same // traffic, this allows NodePort traffic to be forwarded even if the default // FORWARD policy is not accept.

What happened: Network services with heavy load will cause "connection reset" from time to time. Especially those with big payloads. When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. The packet goes back the the original pod, who doesn't recognize the packet because of the wrong source ip, end up RSTing the connection. What you expected to happen: connection not reset How to reproduce it (as minimally and precisely as possible): wrote this just to reproduce the issue: Basically run it in a cluster, and after a while you will be able to see "Connection reset by peer" on both server and client pods. Anything else we need to know?: Related issue: Environment: Kubernetes version (use ): 1.11.7 Cloud provider or hardware configuration: gke (should be general to all provider) OS (e.g: ): PRETTYNAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSIONID="9" VERSION="9 (stretch)" ID=debian HOMEURL="" SUPPORTURL="" BUGREPORTURL="" Kernel (e.g. ): Linux gke-cluster-1-11-default-pool--6nzv 4.14.89+ SMP Wed Jan 9 13:35:00 PST 2019 x86_64 GNU/Linux
/sig network
I created a small app to reproduce this issue:
I've been hitting this exact problem for packets between a Pod and the outside world. So the DROP rule has to go into the INPUT chain. Would that be a job for a CNI driver or the kube-proxy?
was put to iptables mode but not to , right? Isn't ipvs mode affected by this issue, too?
Can anyone point me out how netfilter ignore this STATE invalid packet? I checked the code in netfilter for function iptablenatdo_chain, but didn't find the code that handles the STATE=invalid packet. Thanks for help.
Invalid is connection tracking state. My understanding is: If for some reason conntrack doesn't think it's the packet is "tracked" it could be grouped into "invalid" state. The problem is once it happens, .
ipvs should be fine. do you see similar issue in ipvs?
I also encountered this problem.

Have you solved it
is this a similar issue for ipvs?
The error is a very general networking issue, which could be caused by many different reasons. This specific cause can be reproduced by the It looks like you got an answer already there :)


Question: jobObj, err := createJobWithDefaults(ctx, clientSet, ns.Name, &batchv1.Job{ Spec: batchv1.JobSpec{ CompletionMode: &mode, <del> Completions:    pointer.Int32(500), Parallelism:    pointer.Int32(500), </del> <ins> Completions:    pointer.Int32(100), Parallelism:    pointer.Int32(100), </ins> BackoffLimit:   pointer.Int32(0), }, })

seems we have a problem in branch 1.23, this test has been flaking for some time Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release /sig scheduling /kind flake
/assign
I see the log line for the desired outcome, but after the test has already timed out: And we do see a lot of throttling by APF. I'll just reduce the number of pods the test creates, only in 1.23, as other release seem fine.
just to keep ourselves honest: /remove-sig scheduling :smile:
still seeing this flake in release-1.23 (e.g. 3 runs in a row in , where CI is still running on go1.17)
this single test is failing 100% of runs in the 1.23 PR I'm currently working on
the cherry picks are still open
test has been green since the backports merged


Question: package ipvs import ( <ins> "fmt" netutils "k8s.io/utils/net" </ins> "reflect" "testing" utilipvs "k8s.io/kubernetes/pkg/util/ipvs" utilipvstest "k8s.io/kubernetes/pkg/util/ipvs/testing" <del> netutils "k8s.io/utils/net" </del> ) func Test_GracefulDeleteRS(t *testing.T) {

! no panic Kubernetes version (use ): k8s 1.16
/sig Network
Could you fix this? I think the function needs a lock while being executed.
/triage accepted /good-first-issue todo: add a lock around the body of this function: Similar to
/assign
Oh I see, thank you for letting me know, I'll keep that in mind going forward. :)
Anyone actually on this? It seems like a good fix to have for 23 and maybe backport
I'll have a try.
There is a reproducer here
Oops, I just peeked at the time of the pull request and not realized that you've got such deep conversation.
/remove-good-first-issue /unassign
/assign
hi ! someone work on this now? I would like to try it.
Yes, working on it currently.
are you still planning to give this a shot?
Is there no one working on this? If so, I'd submit a PR for it this week. :)


Question: // 	Marked connection will be drop on INPUT/OUTPUT Chain in filter table // 2. 	In nat table, KUBE-MARK-MASQ rule to mark connections for SNAT // 	Marked connection will get SNAT on POSTROUTING Chain in nat table <del> func (kl *Kubelet) syncNetworkUtil(iptClient utiliptables.Interface) { </del> <ins> func (kl *Kubelet) syncNetworkUtil(iptClient utiliptables.Interface) bool { </ins> // Setup KUBE-MARK-DROP rules dropMark := getIPTablesMark(kl.iptablesDropBit) if _, err := iptClient.EnsureChain(utiliptables.TableNAT, KubeMarkDropChain); err != nil { klog.Errorf("Failed to ensure that %s chain %s exists: %v", utiliptables.TableNAT, KubeMarkDropChain, err) <del> return </del> <ins> return false </ins> } if _, err := iptClient.EnsureRule(utiliptables.Append, utiliptables.TableNAT, KubeMarkDropChain, "-j", "MARK", "--or-mark", dropMark); err != nil { klog.Errorf("Failed to ensure marking rule for %v: %v", KubeMarkDropChain, err) <del> return </del> <ins> return false </ins> } if _, err := iptClient.EnsureChain(utiliptables.TableFilter, KubeFirewallChain); err != nil { klog.Errorf("Failed to ensure that %s chain %s exists: %v", utiliptables.TableFilter, KubeFirewallChain, err) <del> return </del> <ins> return false </ins> } if _, err := iptClient.EnsureRule(utiliptables.Append, utiliptables.TableFilter, KubeFirewallChain, "-m", "comment", "--comment", "kubernetes firewall for dropping marked packets", "-m", "mark", "--mark", fmt.Sprintf("%s/%s", dropMark, dropMark), "-j", "DROP"); err != nil { klog.Errorf("Failed to ensure rule to drop packet marked by %v in %v chain %v: %v", KubeMarkDropChain, utiliptables.TableFilter, KubeFirewallChain, err) <del> return </del> <ins> return false </ins> } // drop all non-local packets to localhost if they're not part of an existing

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: Networking should recreate its iptables rules if they are deleted [Disruptive] Since when has it been failing: Commit Testgrid link: Reason for failure: Anything else we need to know: Possibly related to this? It's listed in the git diff /sig network
/assign /cc Thanks for detecting it so quicklly
most probably is related to
/unassign /assign he has already submitted a fix
/triage accepted
Reviewed. Marking as resolved on the CI Signal Board. Once was merged, failures stopped and they have not re-occurred in the last 8 days (Oct 10th)
Yay


Question: } // rollbackOldManifests rolls back the backuped manifests if something went wrong <del> func rollbackOldManifests(oldManifests map[string]string, origErr error, pathMgr StaticPodPathManager) error { </del> <ins> func rollbackOldManifests(oldManifests map[string]string, origErr error, pathMgr StaticPodPathManager, restoreEtcd bool) error { </ins> errs := []error{origErr} for component, backupPath := range oldManifests { <ins> // Will restore etcd manifest only if it was explicitely requested by setting restoreEtcd to True if component == constants.Etcd && !restoreEtcd { continue } </ins> // Where we should put back the backed up manifest realManifestPath := pathMgr.RealManifestPath(component)

kubeadm etcd upgrade modifying recovery steps /assign /sig cluster-lifecycle /kind bug


Question: package v1 import ( <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/v1beta3" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime" )

has a dependency on pkg/api for , which it embeds in . Is it doing that to take advantage of validation code, or can Config just declare and directly and avoid the dependency?
A quick try (just declaring a in the package) seems to work fine with . I only included those out of habit.
Out of curiosity, is this just for a proper separate of packages or is there some functional issue as well?
mentioned encountering an issue with tests, but it's mostly just for proper separation of packages.
Related: Remove json tags from
See also
Not done:


Question: return forceReplace(f, out, cmd, args, shortOutput, options) } <ins> if cmdutil.GetFlagInt(cmd, "grace-period") >= 0 { return fmt.Errorf("--grace-period must have --force specified") } if cmdutil.GetFlagDuration(cmd, "timeout") != 0 { return fmt.Errorf("--timeout must have --force specified") } </ins> mapper, typer := f.Object(cmdutil.GetIncludeThirdPartyAPIs(cmd)) r := resource.NewBuilder(mapper, typer, resource.ClientMapperFunc(f.ClientForMapping), f.Decoder(true)). Schema(schema).

When replacing an existing pod, the command fails as it tries to create a pod with a name that is still used by terminating pod. How does the option works? deletes a pod (e.g. mypod) creates a new pod with the same name (mypod) What happens? As the command tries to recreate the pod, it fails as the is in state. Thus it can not be created. The problem is the does not wait for a pod to be deleted. Reproducer: run mypod --image=yapei/hello-openshift --generator=run-pod/v1 get pod mypod -o yaml replace -f --force The last command ends with:
option introduce by
With set to positive value, what should be the default behaviour of the command? When set to , should the command hang for and then create the pod? Or is there a way to create a callback (inside a scheduler?) that creates the pod once the current one is deleted?
You could watch for the resource to be deleted and create after you see the deletion happen
have you ever tried to set , I think this will do a non graceful deletion. should make sure the old resource is deleted before it creates a new resource, if not, that's a bug.
Though the help says the is to be used only with , it returns the error above.
Then I think there are two issues we need to track: 1. option should be ignored or just returns error if users are not using option. 2. should always make sure the old resource has been deleted before create the new resource.


Question: kube::test::version::json_object_to_file() { flags=$1 file=$2 <del> kubectl version $flags --output json | sed -e s/'"'/''/g -e s/'}'/''/g -e s/'{'/''/g -e s/'clientVersion:'/'clientVersion:,'/ -e s/'serverVersion:'/'serverVersion:,'/ | tr , 'n' > "${file}" </del> <ins> kubectl version $flags --output json | sed -e s/' '/''/g -e s/'"'/''/g -e s/'}'/''/g -e s/'{'/''/g -e s/'clientVersion:'/'clientVersion:,'/ -e s/'serverVersion:'/'serverVersion:,'/ | tr , 'n' > "${file}" </ins> } kube::test::version::json_client_server_object_to_file() {

I was confused by the that looks like more as machine readable rather than as human readable. I feel that UX could be better if it will be written in multiple lines and without curly braces/quotes:
Agreed on this one, we should pretty-print this output.
I would like to change that tomorrow if you are not working on it.
I'm not working on it, so feel free to provide a PR. But I wouldn't do anything until we get approval from
cc don't some of the tests rely on this output?
They probably do. There should be a option or something that we can use in test infra.
please have a look at: and related issue: The above is approved and had an lgtm labeled, but removed by the bot because of testing flake.
I think we can close this ?
I don't see that it was fixed. Does the output of was improved?
kubectl now has a option which lets you select between YAML or JSON. However, I believe this is reversed. The more human-friendly output is hidden behind an option and the machine-friendly output is the default. I believe it should be the opposite.


Question: } // getContainerNames returns a formatted string containing the container names <del> func getContainerNames(pod *api.Pod) string { </del> <ins> func getContainerNames(containers []api.Container) string { </ins> names := []string{} <del> for _, c := range pod.Spec.Containers { </del> <ins> for _, c := range containers { </ins> names = append(names, c.Name) } return strings.Join(names, " ")

Im extracting logs part from The current behavior is as follow: when pod has one container return logs for this one container when pod has more than one container returns error: Error from server: a container name must be specified for pod kube-dns-v14-z0ho6, choose one of: [kubedns dnsmasq healthz] As you can see in both cases init containers are ignored. The question are: in first case should we return error and list all containers including init containers? in second case should we also list init containers?
first case is designed on purpose that way for the usability, especially the users who switch from docker single container to Kubernetes pod. But what you suggested above for the second case are great, we should do it. Mind to send pr fixing it? Thanks!
sure, I will prepare pr. I'm wondering if we should somehow distinguish init containers and normal containers. we can add to init container name but this may be misleading. we can also add second list with init containers. Something like these: Or just show all in one list without any changes. What do you think?
I prefer not introducing a prefix "init:" here. I like the second option better. cc/ WDYT?
I like the second option as well. We generally avoid prefixes or special syntax except in rare cases.
since I know you are also looking at code in these places.
+1 for the second option.


Question: time_key time path /var/log/containers/*.log pos_file /var/log/gcp-containers.log.pos <del> time_format %Y-%m-%dT%H:%M:%S.%NZ </del> <ins> time_format %Y-%m-%dT%H:%M:%S.%N%Z </ins> tag reform.* read_from_head true </source>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): fluentd timezone timeformat Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): What happened: Log lines from docker have the wrong timestamp. Anything else do we need to know: We are building a custom Docker Image for fluentd/td-agent which is based on your Dockerfile but with some slight changes. One of which is that we set the timezone to something else than UTC. In this case fluentd will produce wrong timestamps for log lines since the string only includes a simple Z in the end. Changing the time_format to fixed the problem for us. Of course this is not really a bug in kubernetes but changing the default to the above will probably help others in the future.
Thanks for reporting! It seems to be a bug


Question: # Logging <ins> **Table of Contents** <!-- BEGIN MUNGE: GENERATED_TOC --> - [Logging](#logging) - [Logging by Kubernetes Components](#logging-by-kubernetes-components) - [Examining the logs of running containers](#examining-the-logs-of-running-containers) - [Cluster level logging to Google Cloud Logging](#cluster-level-logging-to-google-cloud-logging) - [Cluster level logging with Elasticsearch and Kibana](#cluster-level-logging-with-elasticsearch-and-kibana) - [Ingesting Application Log Files](#ingesting-application-log-files) - [Known issues](#known-issues) <!-- END MUNGE: GENERATED_TOC --> </ins> ## Logging by Kubernetes Components Kubernetes components, such as kubelet and apiserver, use the [glog](https://godoc.org/github.com/golang/glog) logging library.  Developer conventions for logging severity are described in [docs/devel/logging.md](../devel/logging.md).

has TODOs for how to bring up cluster addons like monitoring and logging. We should fill these in. requested by user: