Question: } <ins> func Test_Run_UpdateNodeStatusFailBeforeOneVolumeDetachNodeWithReadWriteOnce(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) dsw := cache.NewDesiredStateOfWorld(volumePluginMgr) asw := cache.NewActualStateOfWorld(volumePluginMgr) fakeKubeClient := controllervolumetesting.CreateTestClient() fakeRecorder := &record.FakeRecorder{} fakeHandler := volumetesting.NewBlockVolumePathHandler() ad := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator( fakeKubeClient, volumePluginMgr, fakeRecorder, fakeHandler)) informerFactory := informers.NewSharedInformerFactory(fakeKubeClient, controller.NoResyncPeriodFunc()) nodeLister := informerFactory.Core().V1().Nodes().Lister() nsu := statusupdater.NewFakeNodeStatusUpdater(false /* returnError */) rc := NewReconciler( reconcilerLoopPeriod, maxWaitForUnmountDuration, syncLoopPeriod, false, dsw, asw, ad, nsu, nodeLister, fakeRecorder) reconciliationLoopFunc := rc.(*reconciler).reconciliationLoopFunc() podName1 := "pod-uid1" volumeName := v1.UniqueVolumeName("volume-name") volumeSpec := controllervolumetesting.GetTestVolumeSpec(string(volumeName), volumeName) volumeSpec.PersistentVolume.Spec.AccessModes = []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce} nodeName1 := k8stypes.NodeName("node-name1") dsw.AddNode(nodeName1, false /*keepTerminatedPodVolumes*/) // Add the pod in which the volume is attached to the FailDetachNode generatedVolumeName, podAddErr := dsw.AddPod(types.UniquePodName(podName1), controllervolumetesting.NewPod(podName1, podName1), volumeSpec, nodeName1) if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } // Act reconciliationLoopFunc() // Volume is added to asw, volume should be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, generatedVolumeName, nodeName1, true, asw) // Delete the pod dsw.DeletePod(types.UniquePodName(podName1), generatedVolumeName, nodeName1) // Mock NodeStatusUpdate fail rc.(*reconciler).nodeStatusUpdater = statusupdater.NewFakeNodeStatusUpdater(true /* returnError */) reconciliationLoopFunc() // The first detach will be triggered after at leaset 50ms (maxWaitForUnmountDuration in test). time.Sleep(100 * time.Millisecond) reconciliationLoopFunc() // Right before detach operation is performed, the volume will be first removed from being reported // as attached on node status (RemoveVolumeFromReportAsAttached). After UpdateNodeStatus operation which is expected to fail, // controller then added the volume back as attached. // verifyVolumeReportedAsAttachedToNode will check volume is in the list of volume attached that needs to be updated // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, generatedVolumeName, nodeName1, true, asw) } </ins> func Test_Run_OneVolumeDetachFailNodeWithReadWriteOnce(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t)

Recently, the cpu load of my kube-apiserver gets higher after I expanded more pods and nodes into the cluster. And I found that some volumes declared by pods could not be mounted after the restart of pods, even though the pods were successful scheduled. here's the kubectl describe information: After checking the logs of attachdetachcontroller, I found that in , return an error, thus the was skipped. That should be related to the high load on my kube-apiserver as it occasionally returns 500 http code (but this may not be the root cause). Here is the error message: Here comes the problem: In my opinion, the next loop of would retry and , since the state of actualStateOfWorld is different from the state of desiredStateOfWorld. but it didn't happened. By recording some additional logs, I notice that the pod was scheduled to the destination node by the scheduler right after above error message appeared, which cause executed. That would write the volume info back to . After that, the next loop of would skip the detach operation for that volume, since desiredStateOfWorld was 'equal to' actualStateOfWorld. The log snippet is as follows: Finally, attachdetachcontroller would not attach the volume neither, because the states of desiredStateOfWorld and actualStateOfWorld are the same. attachdetachcontroller mistakenly thought the volume was attached, but it' not the case. when fail, attachdetachcontroller should retry to detach the volume until the the detach operation completed. However, it was interrupted by the update of desiredStateOfWorld. a log and before a pod mounting pv Then you'll found log message "force skip detaching" would disappear after a few loops of No response <details</details<detailsN/A </detailsNo response <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig storage
If fails, that means the volume has not been detached and the volumeattachment is still on the nodeStatus, then kubelet should see that the volume still attaches to the node, so I don't understand why your pod failed to mount, it may not be caused by the controller. what's the error for kubelet?
! There are . The one before was failed, but another which is at the end of was successfully executed. Thus it cause the kubelet coundn't get the volume in .
kubelet log repeat the message below infinitely:
Okay, I know what you said, actually this problem is similar with the one which was fixed by , but it is not completely. Give another pr below to fix this.
Can we have the fix () back-ported to 1.21? Given that this completes the , the initial fix for which was , it would make sense to have this follow-up fix also be to 1.21. Looks like there is a that we could get this into. We have a 100+ clusters deployed in production that are running 1.21. We see this bug at least once a week and it's a headache to deal with. Upgrading to 1.22 or higher is a big undertaking for us.


Question: } ginkgo.By("Deleting the previously created pod") <del> err = e2epod.DeletePodWithWait(ctx, m.cs, pod) framework.ExpectNoError(err, "while deleting pod for resizing") </del> <ins> if test.simulatedCSIDriverError == expansionFailedMissingStagingPath { e2epod.DeletePodOrFail(ctx, m.cs, pod.Namespace, pod.Name) } else { err = e2epod.DeletePodWithWait(ctx, m.cs, pod) framework.ExpectNoError(err, "while deleting pod for resizing") } </ins> ginkgo.By("Creating a new pod with same volume") pod2, err := m.createPodWithPVC(pvc)

When a pod is quickly deleted and re-created on a node, it may not be necessary to unstage the volume and in which case only node-publish is called for the volume. When this happens - stagepath won't be set for RPC call because volume was not staged even though, volume does have capability. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. Quickly delete and recreate a pod No response master Any No response_ <details</details<details</details<details</details
/sig storage /triage accepted


Question: }{ "catch all host and default backend with default TLS returns empty TLS": { rules: []string{ <del> "_/=catchall:8080,tls=", </del> <ins> "/=catchall:8080,tls=", </ins> }, ingressclass:   ingressClass, defaultbackend: "service1:https",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl create ingress panics because there is no cmdutil.AddDryRunFlag What you expected to happen: Command to work How to reproduce it (as minimally and precisely as possible): Compile the upstream, run ` Anything else we need to know?: Environment: Kubernetes version (use ): upstream/dev /sig cli /kind bug /priority critical-urgent
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.Failf("didn't find resource %s in the discovery doc", spec.Names.Plural) } }) <ins> ginkgo.It("should validate PreferredVersion for each APIGroup", func() { // get list of APIGroup endpoints list := &metav1.APIGroupList{} err := f.ClientSet.Discovery().RESTClient().Get().AbsPath("/apis/").Do(context.TODO()).Into(list) framework.ExpectNoError(err, "Failed to find /apis/") framework.ExpectNotEqual(len(list.Groups), 0, "Missing APIGroups") for _, group := range list.Groups { framework.Logf("Checking APIGroup: %v", group.Name) // locate APIGroup endpoint checkGroup := &metav1.APIGroup{} apiPath := "/apis/" + group.Name + "/" err = f.ClientSet.Discovery().RESTClient().Get().AbsPath(apiPath).Do(context.TODO()).Into(checkGroup) framework.ExpectNoError(err, "Fail to access: %s", apiPath) framework.ExpectNotEqual(len(checkGroup.Versions), 0, "No version found for %v", group.Name) framework.Logf("PreferredVersion.GroupVersion: %s", checkGroup.PreferredVersion.GroupVersion) framework.Logf("Versions found %v", checkGroup.Versions) // confirm that the PreferredVersion is a valid version match := false for _, version := range checkGroup.Versions { if version.GroupVersion == checkGroup.PreferredVersion.GroupVersion { framework.Logf("%s matches %s", version.GroupVersion, checkGroup.PreferredVersion.GroupVersion) match = true break } } framework.ExpectEqual(true, match, "failed to find a valid version for PreferredVersion") } }) </ins> })

] 20 July 2020 [x] Two weeks soak end date: 3 August 2020 According to this APIsnoop query, there are still an endpoint which is untested. Get a list of all group /apis Iterate through list Query each endpoint List the PreferredVersion for the endpoint Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 15 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/retitle Write checkAPIGroupPreferredVersion Test +Promote - +16 Endpoint coverage
/reopen Merged /assign Ready for creation of Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged /assign Ready for creation of Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: wantStatusFailed:    0, wantStatusSucceeded: 0, }, <ins> "ignore pod failure based on OnPodConditions, ignored failures delays pod recreation": { enableJobPodFailurePolicy: true, job: batch.Job{ TypeMeta:   metav1.TypeMeta{Kind: "Job"}, ObjectMeta: validObjectMeta, Spec: batch.JobSpec{ Selector:     validSelector, Template:     validTemplate, Parallelism:  pointer.Int32(1), Completions:  pointer.Int32(1), BackoffLimit: pointer.Int32(0), PodFailurePolicy: &batch.PodFailurePolicy{ Rules: []batch.PodFailurePolicyRule{ { Action: batch.PodFailurePolicyActionIgnore, OnPodConditions: []batch.PodFailurePolicyOnPodConditionsPattern{ { Type:   v1.DisruptionTarget, Status: v1.ConditionTrue, }, }, }, }, }, }, }, pods: []v1.Pod{ { ObjectMeta: metav1.ObjectMeta{ DeletionTimestamp: &now, }, Status: v1.PodStatus{ Phase: v1.PodFailed, Conditions: []v1.PodCondition{ { Type:   v1.DisruptionTarget, Status: v1.ConditionTrue, }, }, }, }, }, wantConditions:      nil, wantStatusActive:    0, wantStatusFailed:    0, wantStatusSucceeded: 0, }, </ins> "fail job based on OnPodConditions": { enableJobPodFailurePolicy: true, job: batch.Job{

Pod backoff calculation excludes pods that match an rule in the failure policy: This could lead to fast retries, overwhelming the control plane, if most failures match an rule. The semantics of is not to count towards the limit, but the backoff should still be respected. /sig apps Recreations to be throttled. a Job with failure policy that matches every kind of failure. a container that always fails You would observe immediate recreation. No response_ 27+ <detailsN/A </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc


Question: // UpdateLoadBalancer updates hosts under the specified load balancer. func (az *Cloud) UpdateLoadBalancer(ctx context.Context, clusterName string, service *v1.Service, nodes []*v1.Node) error { <ins> if !az.shouldUpdateLoadBalancer(clusterName, service) { klog.V(2).Infof("UpdateLoadBalancer: skipping service %s because it is either being deleted or does not exist anymore", service.Name) return nil } </ins> _, err := az.EnsureLoadBalancer(ctx, clusterName, service, nodes) return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The issue has been observed on Azure but should also apply to other providers as well. On Azure we observed that there are orphaned Public IPs that don't belong to any Kubernetes LoadBalancer service but from looking at the tags and the name it is clear that Kubernetes created them. In the Azure Activity Log, you can see that the Public IP was deleted when the K8s service was deleted but recreated a few minutes later. That re-creation is also visible in the K8s controller manager logs as a reconcile of the service. In our case, we add additional worker nodes and delete services at the same time. I think this leads to the re-creation here Because the service list is retrieved at the beginning of the reconcile loop, services that are deleted while the loop is reconciling a recreated by that loop and afterwards orphaned. What you expected to happen: No orphaned public IPs How to reproduce it (as minimally and precisely as possible): Create a few Kubernetes Load Balancer services (we have around 20). Then scale your worker nodes and delete one service at the same time. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0 Cloud provider or hardware configuration: Azure OS (e.g: ): Core OS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
Thanks for reporting the issue. I think this should be fixed during v1.16 release cycle. /milestone v1.16
In the nodeSyncLoop -updateLoadBalancerHosts -lockedUpdateLoadBalancerHosts call chain, I don't see where the mutex lock is obtained.
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 4 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged.

Is this issue still planned to be fixed for the 1.16 release?
The issue would probably not able to be fixed in v1.16. AFAIK, no one is working on it now.
Thank you for the update! I'll go ahead and remove the issue from the milestone. If anything changes in the meanwhile, please let us know. /milestone clear
/assign
/assign
GitHub didn't allow me to assign the following users: chewong. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Doesn't LB Finalizer support fix this? We enabled this by default starting v1.16
Initial PR: Bump to beta in v1.16:


Question: pvs := make([]*v1.PersistentVolume, maxPVs) for i := 0; i < maxPVs; i++ { // This PV will be claimed, released, and deleted <del> pvs[i] = createPV("pv-"+strconv.Itoa(i), "/tmp/foo"+strconv.Itoa(i), strconv.Itoa(i)+"G", </del> <ins> pvs[i] = createPV("pv-"+strconv.Itoa(i), "/tmp/foo"+strconv.Itoa(i), strconv.Itoa(i+1)+"G", </ins> []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce}, v1.PersistentVolumeReclaimRetain) }

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If someone requests a zero (0) capacity PVC and an administrator creates a zero (0) capacity PV, Kubernetes will happily bind them together. Unfortunately, any use of this PVC/PV combination by a pod will fail. It fails because the zero capacity PV cannot be formatted or mounted successfully. Note: The same issue happens for dynamically provisioned PVs. What you expected to happen: I expect that Kubernetes would not allow creation of invalid PVC and PV objects, aka zero (0) capacity PVs and PVCs. How to reproduce it (as minimally and precisely as possible): Create a pod that requests a zero capacity PVC, using dynamic PV provisioning is simpler than creating your own zero (0) capacity PV. The pod will fail to reach a running state, since the related PV fails to be formatted and mounted successfully. Here is the error that I received for a test I performed requesting a zero capacity PVC using ceph rbd: Anything else we need to know?: Here is a related item with a specific fix for external ceph rbd provisioner: Environment: Nothing specific to environment
/sig storage
/assign


Question: // Cache may be overridden by API object with higher version, compare but ignore resource version. newCachedPVC := cachedPVC.DeepCopy() newCachedPVC.ResourceVersion = pvc.ResourceVersion <del> if !reflect.DeepEqual(newCachedPVC, pvc) { t.Errorf("cached PVC check failed [A-expected, B-got]:n%s", diff.ObjectDiff(pvc, cachedPVC)) </del> <ins> if diff := cmp.Diff(pvc, newCachedPVC); diff != "" { t.Errorf("cached PVC check failed (-want, +got):n%s", diff) </ins> } }

When scheduling pod, we encountered the following event , then the scheduler will backoff to reschedule again, but this add more waiting time. From the event message, we can see it is the PVC update conflict leads to the , so need to add retry logic on PVC updating.### Need to add retry logic on PVC updating, to avoid the backoff, to improve the scheduling efficiency. a PVC and then create Pod; scheduing; the PVC annotation or label before the Pod scheduled success; will encounter this issue. No response <details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage note that this is not a supported client/server skew: Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.4-tke.23", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-03-15T09:25:31Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
/sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: continue } if strings.ToUpper(*size.Name) == vmsize { <del> klog.V(2).Infof("got a matching size in getMaxDataDiskCount, Name: %s, MaxDataDiskCount: %d", *size.Name, *size.MaxDataDiskCount) </del> <ins> klog.V(12).Infof("got a matching size in getMaxDataDiskCount, Name: %s, MaxDataDiskCount: %d", *size.Name, *size.MaxDataDiskCount) </ins> return int64(*size.MaxDataDiskCount) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There are too many logs in GetVolumeLimits on azure disk, which would flush kubelet logs, it happens every 10 seconds: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /assign /sig azure


Question: // Volume information is extracted from the directory structure: // (ROOT_DIR)/(POD_ID)/volumes/(VOLUME_KIND)/(VOLUME_NAME) for _, podIDDir := range podIDDirs { <ins> if !podIDDir.IsDir() { continue } </ins> podID := podIDDir.Name() podIDPath := path.Join(mountPath, podID, "volumes") volumeKindDirs, err := ioutil.ReadDir(podIDPath)

This error appears every 10 seconds (with .bashlogout & .bashprofile also), when set up with vagrant: Aug 12 23:16:58 kubernetes-minion-1 kubelet[]: E0812 23:16:58. ] Could not read directory: , (open not a directory) This is because the kubelet root directory defaults to /var/lib/kubelet, which is also the homedir for the kubelet user, and so the user's default files trip up kubelet. Manually deleting the files (rm *) causes the errors to stop without any obvious ill-effects.
fixed in


Question: func (ev *Evaluator) prepareCandidate(ctx context.Context, c Candidate, pod *v1.Pod, pluginName string) *framework.Status { fh := ev.Handler cs := ev.Handler.ClientSet() <del> for _, victim := range c.Victims().Pods { </del> <ins> ctx, cancel := context.WithCancel(ctx) defer cancel() errCh := parallelize.NewErrorChannel() preemptPod := func(index int) { victim := c.Victims().Pods[index] </ins> // If the victim is a WaitingPod, send a reject message to the PermitPlugin. // Otherwise we should delete the victim. if waitingPod := fh.GetWaitingPod(victim.UID); waitingPod != nil {

When a pod requires preempting more than one pod, this happens sequentially. It should be done in parallel. /sig scheduling /sig scalability Preemption happens in the scheduling cycle, meaning that no other scheduling attempts happen while preemption is ongoing, hurting throughput. An alternative is to the deletes in a separate routine, like we do for binding.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I'm a newbie, may I have a try at this proposal?
Yes, this is a good candidate for new contributors. However, I would like to hear from SIG scalability whether there are any concerns about this.
Parallelizing the loops looks like the easier option to implement, but deleting in a separate routine will likely offer the better throughput for the average case.
Parallelizing loops seems fine from SIG Scalability perspective - we already do something similar for example in KCM when creating pods or in scheduler each binding is done by separate goroutine. One question though, is there a limit of pods that we want to preempt here? 100 parallel calls will be probably fine, but 1000 wouldn't be most likely.
We only preempt pods from a single node, so that's a natural threshold. And we have a parallelizing mini-library that has 16 workers by default.
Hi, do you still want to work on this one?
YesIt's still under discussion.
So we can use the parallelizing mini-library to do this?
You can use this parallelizer we defined in scheduler framework directly.
feel free to take it by writing on a comment.
/assign


Question: epIpt := netlinktest.ExpectedIptablesChain{ string(kubeServicesChain): {{ JumpChain: string(KubeLoadBalancerChain), MatchSet: kubeLoadBalancerSet, <ins> }, { JumpChain: string(KubeMarkMasqChain), MatchSet: kubeClusterIPSet, }, { JumpChain: string(KubeNodePortChain), MatchSet: "", }, { JumpChain: "ACCEPT", MatchSet: kubeClusterIPSet, }, { JumpChain: "ACCEPT", MatchSet: kubeLoadBalancerSet, </ins> }}, string(kubeLoadBalancerSet): {{ JumpChain: string(KubeMarkMasqChain), MatchSet: "",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --ipvs mode didn't preserve client IP for NodePort Service with protocol SCTP and ExternalTrafficPolicy Local. iptables mode works fine. ipvs mode with TCP and UDP protocol works fine. Client IP for NodePort Service should be preseved when ExternalTrafficPolicy is Local regardless of the mode and the protocol. kube-proxy mode to ipvs a NodePort service with protocol SCTP and ExternalTrafficPolicy Local the NodePort service, the client IP was not preserved This is caused by wrong order of iptables rules that match kubeNodePortLocalSetSCTP and kubeNodePortSetSCTP: Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /assign
/triage accepted


Question: StartedInitContainersErrorCount int64 } <ins> // ensurePodsStatsOnNode returns a boolean after checking Pods statistics on a particular node func ensurePodsStatsOnNode(ctx context.Context, c clientset.Interface, namespace, nodeName string, statsChecked *bool) (bool, error) { nodeStats, err := e2ekubelet.GetStatsSummary(ctx, c, nodeName) framework.ExpectNoError(err, "Error getting node stats") for _, podStats := range nodeStats.Pods { if podStats.PodRef.Namespace != namespace { continue } // check various pod stats if *podStats.CPU.UsageCoreNanoSeconds <= 0 { framework.Logf("Pod %s/%s stats report cpu usage equal to %v but should be greater than 0", podStats.PodRef.Namespace, podStats.PodRef.Name, *podStats.CPU.UsageCoreNanoSeconds) return false, nil } if *podStats.Memory.WorkingSetBytes <= 0 { framework.Logf("Pod %s/%s stats report memory usage equal to %v but should be greater than 0", podStats.PodRef.Namespace, podStats.PodRef.Name, *podStats.CPU.UsageCoreNanoSeconds) return false, nil } for _, containerStats := range podStats.Containers { *statsChecked = true // check various container stats if *containerStats.CPU.UsageCoreNanoSeconds <= 0 { framework.Logf("Container %s stats report cpu usage equal to %v but should be greater than 0", containerStats.Name, *containerStats.CPU.UsageCoreNanoSeconds) return false, nil } if *containerStats.Memory.WorkingSetBytes <= 0 { framework.Logf("Container %s stats report memory usage equal to %v but should be greater than 0", containerStats.Name, *containerStats.Memory.WorkingSetBytes) return false, nil } if *containerStats.Logs.UsedBytes <= 0 { framework.Logf("Container %s stats log usage equal to %v but should be greater than 0", containerStats.Name, *containerStats.Logs.UsedBytes) return false, nil } } } return true, nil } </ins> // getCurrentHostProcessMetrics returns a HostPRocessContainersMetrics object. Any metrics that do not have any // values reported will be set to 0. func getCurrentHostProcessMetrics(ctx context.Context, f *framework.Framework, nodeName string) (HostProcessContainersMetrics, error) {

ci-kubernetes-e2e-capz-master-containerd-windows Kubernetes e2e suite: [It] [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers container stats validation feb 8th 2023 reading of the log is showing reporting zero It looks like the pod had started up just as it check the failure on the test. Adding a retry or a longer wait might make the test more reliable /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: go_test( name = "go_default_test", <del> srcs = ["humanreadable_test.go"], </del> <ins> srcs = [ "humanreadable_test.go", "template_test.go", ], </ins> embed = [":go_default_library"], deps = [ "//pkg/apis/core:go_default_library",

Currently if one wants to view secrets stored in k8s they must follow the process described here: This involves copy&pasting. It seems to me that kubectl could provide a more helpful interface to base64 decrypt the users on behalf of the user saving them the trouble.
The kubelet does decode secrets before exposing them to the pod. Also the title doesn't really make sense. Secrets aren't encrypted so there's nothing to decrypt.
/kind feature
Yes, though this is low priority since we mostly don't want end users to view these once they get into the system.
Why can't that be an API permissions issue? I mean, if I really want to see it, I can just launch a pod to tell me.
I've update the issue title with what I understand as the request. To the best of my understanding this is just a request for us to avoid the need to pipe into base64 when inspecting secrets.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
I would like to try to make this work before it rots. cc Any guidance on a preferred implementation?
What do you think about adding support for functions in templates? This would allow someone to use the existing and a template that uses the function -- or a wide variety of other interesting functions as a side effect. I believe (but have not yet tested) that the following might be sufficient to enable the requested behaviour (plus the change to the dependencies list, of course):
Hi I suppose my thoughts on this issue have changed a little since I wrote it. The situation isn't quite as bad as I described as it is possible to decode a secret in one go without then need to copy and paste. i.e.

That said, that is a bit of a mouthful. Also, it might be nice to more easily 'work' with individual secrets and set them as well as get them. Should a user really need to pull down an entire group of secrets and edit a file just to be able to update a single password? For example something like this might be possible: What do people think? PS: Should this have been an issue on [kubectl] instead? PPS: I think some of the ugliness of this comes from configmaps being very similar to secrets except for the base64 encoding. Have there been any thoughts around bringing them into line? If memory serves, it might actually be nicer, due to YAML syntax-shenanigans, if configmaps were base64 encoded too.) [kubectl]:
Hi ... yes, I was thinking that it would be handy to be able to do something like: which is essentially the same as what you did without the overhead of the base64 process, or embed the information in a larger template like or in theory be able to use any of the other Sprig functions if someone found them useful. I think that having the Sprig methods available for generic formatting would just be useful on its own, really. On the topic of a or a , that seems pretty cool. There seem to be quite a lot of options for that I think people might expect to see mirrored in a if it were to exist. Thoughts?
It's also come to my attention that works great on MacOS but on Linux it's , so if someone, say, has a Helm chart where the notes tell the user to retrieve the secret data using one or the other of these, it won't work on the other one. would clearly be a simpler experience for the user than Hmm... thinking further: does does the idea conflict with the documentation in ? Perhaps we'd have to do something like: or similar?
/remove-lifecycle rotten
Small tip: you can use which works on both Linux and Mac.
Thank you
Update: I've looked at the huge amount of additional code that would be imported as a result of importing sprig and decided to go with a much smaller change instead, which is simply to implement a template function using . See PR with code and test.
please see the linked PR which addresses the original issue in a minimal fashion -- it simply provides a template function.

It has passed all the tests and needs review; if it's good enough to solve the original problem I'm happy to squash the commits and can hope to get it merged. As far as solving evolved description of the problem, I'm not sure where to start or if you really want me to go in that direction. Would you consider taking this change and splitting the additional work for being able to get / set the contents of an individual key in a secret into a separate issue? If you have any guidance on what you'd like to see with that I'd be happy to take a look, but it might be out of my current grasp.


Question: github.com/containerd/go-runc => github.com/containerd/go-runc v0.0.0-20180907222934-5a6d9f37cfa3 github.com/containerd/ttrpc => github.com/containerd/ttrpc v1.0.2 github.com/containerd/typeurl => github.com/containerd/typeurl v1.0.1 <del> github.com/containernetworking/cni => github.com/containernetworking/cni v0.8.0 </del> <ins> github.com/containernetworking/cni => github.com/containernetworking/cni v0.8.1 </ins> github.com/coredns/corefile-migration => github.com/coredns/corefile-migration v1.0.11 github.com/coreos/bbolt => github.com/coreos/bbolt v1.3.2 github.com/coreos/etcd => github.com/coreos/etcd v3.3.13+incompatible

is identified with with severity rating as high. This is found using snyk scan while running against k/k master We're not using directly however, it gets tagged for vulnerability scans as we use . The fix is already present in and we can bump the dep. Bump dep to which has the mentioned CVE fix. and check Kubernetes version (use ): n/a Cloud provider or hardware configuration: any OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a /sig architecture /area code-organization for the dependency
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: logEntryMessageRegex = regexp.MustCompile("(?:Id+ d+:d+:d+.d+       d+ logs_generator.go:67] )?(d+) .*") ) <del> // Type to track the progress of logs generating pod type loggingPod struct { // Name of the pod Name string // Cache of ingested and read entries Occurrences map[int]logEntry // Number of lines expected to be ingested from this pod ExpectedLinesNumber int } </del> type logEntry struct { Payload string } <ins> func (entry logEntry) getLogEntryNumber() (int, bool) { submatch := logEntryMessageRegex.FindStringSubmatch(entry.Payload) if submatch == nil || len(submatch) < 2 { return 0, false } lineNumber, err := strconv.Atoi(submatch[1]) return lineNumber, err == nil } </ins> type logsProvider interface { Init() error Cleanup()

Pods should be run in parallel with checking the logs from the backend to fit into the timeout.


Question: // Use metadata server if possible if g.isCurrentInstance(instanceName) { <del> internalIP, err := metadata.Get("instance/network-interfaces/0/ip") </del> <ins> nics, err := metadata.Get("instance/network-interfaces/") </ins> if err != nil { <del> return nil, fmt.Errorf("couldn't get internal IP: %v", err) </del> <ins> return nil, fmt.Errorf("couldn't get network interfaces: %v", err) </ins> } <del> externalIP, err := metadata.Get("instance/network-interfaces/0/access-configs/0/external-ip") if err != nil { return nil, fmt.Errorf("couldn't get external IP: %v", err) } addresses := []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: internalIP}, {Type: v1.NodeExternalIP, Address: externalIP}, </del> <ins> nicsArr := strings.Split(nics, "/n") nodeAddresses := []v1.NodeAddress{} for _, nic := range nicsArr { if nic == "" { continue } internalIP, err := metadata.Get(fmt.Sprintf(networkInterfaceIP, nic)) if err != nil { return nil, fmt.Errorf("couldn't get internal IP: %v", err) } nodeAddresses = append(nodeAddresses, v1.NodeAddress{Type: v1.NodeInternalIP, Address: internalIP}) acs, err := metadata.Get(fmt.Sprintf(networkInterfaceAccessConfigs, nic)) if err != nil { return nil, fmt.Errorf("couldn't get access configs: %v", err) } acsArr := strings.Split(acs, "/n") for _, ac := range acsArr { if ac == "" { continue } externalIP, err := metadata.Get(fmt.Sprintf(networkInterfaceExternalIP, nic, ac)) if err != nil { return nil, fmt.Errorf("couldn't get external IP: %v", err) } if externalIP != "" { nodeAddresses = append(nodeAddresses, v1.NodeAddress{Type: v1.NodeExternalIP, Address: externalIP}) } } </ins> } <del> if internalDNSFull, err := metadata.Get("instance/hostname"); err != nil { </del> <ins> internalDNSFull, err := metadata.Get("instance/hostname") if err != nil { </ins> klog.Warningf("couldn't get full internal DNS name: %v", err) } else { <del> addresses = append(addresses, </del> <ins> nodeAddresses = append(nodeAddresses, </ins> v1.NodeAddress{Type: v1.NodeInternalDNS, Address: internalDNSFull}, v1.NodeAddress{Type: v1.NodeHostName, Address: internalDNSFull}, ) } <del> return addresses, nil </del> <ins> return nodeAddresses, nil </ins> } }

What happened: when initializing a kubernetes cluster in Google Cloud manually (not using GKE), internal (and external if available/applicable) node IP is not populated in "kubectl get nodes" if the interface used to connect to the master and passed to kubelet does not correspond to the first network interface, nic0 on google cloud console UI. I cannot "kubectl edit node" and patch in the node IP information into the "addresses" field either. This issue does not occur when kubelet --node-ip=(ip of nic0). kubelet on all members of the cluster declare cloud-provider="gce". What you expected to happen: when kubelet is given a node-ip argument, it is checked against all nics of the google cloud instance and populated in node information even if a nic other than nic0 is used. Or at least I am allowed to override nodeIP somehow after the node is brought up. How to reproduce it (as minimally and precisely as possible): We have scripts that automate this "manual kubernetes" process, I will describe it as generally as I can but i can share chunks of the script as needed. Given two VPCs, initialize an instance for master, one nic on VPC1 initialize an instance for worker with two nics, one on VPC2, the other on VPC1. install required dependencies on both (apt-get docker-ce kubelet kubeadm kubectl ...) on master, mkdir -p /etc/default/kubelet; echo "KUBELETEXTRAARGS=--cloud-provider='gce' /etc/default/kubelet; kubeadm init on worker, mkdir -p /etc/default/kubelet; echo "KUBELETEXTRAARGS=--node-ip=(nic1 ip) --cloud-provider='gce' /etc/default/kubelet; kubeadm join wait a bit. kubectl get nodes -o wide, verify no internal IP or external IP information is populated for the worker journalctl -xef on worker to see something like ] Failed to set some node status fields: failed to get node address from cloud provider that matches ip: (nic1 ip) Anything else we need to know?: i'm guessing the problem stems from something like where we only check nic0 and return that, and then this bit here only loops through nic0's info, doesn't match, and then we return an error.

if the first bit returned an array of IPs instead of just a single set, that would probably fix this. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Google Cloud, all n1-standard-4 instances OS (e.g: ): NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 18.04.4 LTS" VERSIONID="18.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=bionic UBUNTUCODENAME=bionic Kernel (e.g. ): Linux 5.0.0-1033-gcp -Ubuntu SMP Tue Mar 3 04:36:26 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: installed kubeadm kubelet docker and whatever else through apt-get Network plugin and version (if this is a network-related bug): Others:
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider ? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
w.r.t. NodeAddresses in We can keep calling Get for 1/ip, 2/ip, ... until err is non-nil. Is there a better way to find how many interfaces there are ?
i haven't used the gcloud go metadata client, but looping until error or hitting instance/network-interfaces directly for the interface indices available is probably about the same since NIC count is limited by VM tier anyways.


Question: failedStatus.Phase = v1.PodFailed tests := []struct { <del> probe          v1.Probe podStatus      *v1.PodStatus expectContinue bool expectSet      bool expectedResult results.Result </del> <ins> probe                v1.Probe podStatus            *v1.PodStatus expectContinue       map[string]bool expectSet            bool expectedResult       results.Result setDeletionTimestamp bool </ins> }{ { // No status. <del> expectContinue: true, </del> <ins> expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> }, { // Pod failed podStatus: &failedStatus, }, <ins> { // Pod deletion podStatus:            &runningStatus, setDeletionTimestamp: true, expectSet:            true, expectContinue: map[string]bool{ readiness.String(): true, }, expectedResult: results.Success, }, </ins> { // No container status <del> podStatus:      &otherStatus, expectContinue: true, </del> <ins> podStatus: &otherStatus, expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> }, { // Container waiting <del> podStatus:      &pendingStatus, expectContinue: true, </del> <ins> podStatus: &pendingStatus, expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> expectSet:      true, expectedResult: results.Failure, },

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: "File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated "+ "after server cert). If HTTPS serving is enabled, and --tls-cert-file and "+ "--tls-private-key-file are not provided, a self-signed certificate and key "+ <del> "are generated for the public address and saved to /var/run/kubernetes.") </del> <ins> "are generated for the public address and saved to the directory specified by --cert-dir.") </ins> fs.StringVar(&s.ServerCert.CertKey.KeyFile, "tls-private-key-file", s.ServerCert.CertKey.KeyFile, "File containing the default x509 private key matching --tls-cert-file.")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Not all variants of api servers store the TLS cert files at /var/run/kubernetes. After the commit was merged back in Feb, some variants such as kube-aggregator stores them at '', but the hard-coded help message still says it will save the keys and certs generated into /var/run/kubernetes. What you expected to happen: Either we refactor the code so the help message prints an accurate location, or we change the help message to make it always true. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: No. Environment: Kubernetes version (use ): at least 1.7 and above Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
/sig api-machinery
pls help confirm.
We should make the messages accurate.
/assign
GitHub didn't allow me to assign the following users: jennybuckley. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: framework.ExpectNoError(err, "Fluentd deployed incorrectly") By("Running synthetic logger") <del> pod := createLoggingPod(f, podName, "", 10*60, 10*time.Minute) </del> <ins> pod := startNewLoggingPod(f, podName, "", 10*60, 10*time.Minute) </ins> defer f.PodClient().Delete(podName, &meta_v1.DeleteOptions{}) err = framework.WaitForPodNameRunningInNamespace(f.ClientSet, podName, f.Namespace.Name) framework.ExpectNoError(err, fmt.Sprintf("Should've successfully waited for pod %s to be running", podName))

Pods should be run in parallel with checking the logs from the backend to fit into the timeout.


Question: framework.Logf("Deleting Gluster endpoints %q...", name) err := cs.CoreV1().Endpoints(ns.Name).Delete(name, nil) if err != nil { <del> framework.Failf("Gluster delete endpoints failed: %v", err) </del> <ins> if !errors.IsNotFound(err) { framework.Failf("Gluster delete endpoints failed: %v", err) } framework.Logf("Gluster endpoints %q not found, assuming deleted", name) </ins> } framework.Logf("Deleting Gluster server pod %q...", g.serverPod.Name) err = framework.DeletePodWithWait(f, cs, g.serverPod)

Failing on gke only: https://k8s- https://k8s-
/assign
Unfortunately the exact error message that its failing with is lost due to a typo: I have fixed that typo locally and am running the tests repeatedly to try to reproduce the flake. have been unable to reproduce the flake on HEAD of Kubernetes in GKE in 15 runs so far. Running 150 tests overnight.
Unable to repro on my own GKE cluster. Merged the error message fix, will monitor for the next flake to debug more


Question: kube::test::get_object_assert 'rc busybox1' "{{$rc_replicas_field}}" '1' # Command output_message=$(! kubectl scale --current-replicas=1 --replicas=2 -f hack/testdata/recursive/rc --recursive 2>&1 "${kube_flags[@]}") <del> # Post-condition: busybox0 & busybox replication controllers are scaled to 2 # replicas, and since busybox2 is malformed, it should error </del> <ins> # Post-condition: busybox0 & busybox1 replication controllers are scaled to 2 replicas, and since busybox2 is malformed, it should error </ins> kube::test::get_object_assert 'rc busybox0' "{{$rc_replicas_field}}" '2' kube::test::get_object_assert 'rc busybox1' "{{$rc_replicas_field}}" '2' kube::test::if_has_string "${output_message}" "Object 'Kind' is missing"

cc
Assigning to per test-
Flaked again
The flake actually happened when pausing the deployment. Changing issue title.
When , sometimes one of the deployments isn't paused. cc
I can't access the internal Google logs to examine the cause, nor can I reproduce this issue locally. Is there any additional information that you can provide to help troubleshoot the flake?
I've gone ahead and attempted a fix for this in
We should fix this in eventually
the external links are here ():


Question: fmt.Fprint(cmdOut, "Creating federation control plane service...") glog.V(4).Info("Creating federation control plane service") <del> svc, ips, hostnames, err := createService(cmdOut, hostClientset, i.commonOptions.FederationSystemNamespace, serverName, i.commonOptions.Name, i.options.apiServerAdvertiseAddress, i.options.apiServerServiceType, i.options.dryRun) </del> <ins> svc, ips, hostnames, err := createService(cmdOut, hostClientset, i.commonOptions.FederationSystemNamespace, serverName, i.commonOptions.Name, i.options.apiServerAdvertiseAddress, i.options.apiServerNodePortPortPtr, i.options.apiServerServiceType, i.options.dryRun) </ins> if err != nil { return err }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS using Juju OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ): 4.4.0-59-generic Install tools: Juju Others: What happened: kubefed init entered a infinite loop health checking the federation api service What you expected to happen: kubefed to exit with a success code How to reproduce it (as minimally and precisely as possible): Anything else we need to know: When doing a kubefed init on a cluster setup using VMs on aws (using Juju in my case) the worker nodes have two IPs: one that is visible to the internet and one internal visible only to VMs within aws. In my case for example the VM on aws hosting the kubernetes-worker node had external IP in the 54.174.73.1/24 subnet and it also had an internal IP in the 172.31.60.1/24 subnet. Using a client machine outside aws to bootstrap federation: I got the kubefed client stuck on doing a health check against the clouds internal subnet https://172.31.60.XXX:/healthz . To address this I can use the flag and pass the external IP of any kubernetes-worker node.

With this flag kubeconfig file gets updated with the right endpoint for the federation-apiserver. However, apart from the I should also be able to specify the desired port the service will be listening on. I have to make sure that port is opened on the aws routing/firewall prior to issuing . In the example above, kubefed init will try to reach https://54.174.73.XXX:/healthz, port is randomly assigned and is not reachable. I should be able to first open the port on aws and then do a with and port pinned to .
/cc
/assign
is there a reason why you are not using the default load balanced service on AWS?
I am using Juju to stay cloud agnostic. AWS just happened to be where I deployed this time. Juju just grabs machines (virtual or physical) from wherever you have resources from. In effect your cluster is more like an on-prem than anything else. How would the default load balanced service work?
got it.


Question: kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <del> ### Rollback a deployment </del> <ins> ### Rollout on multiple deployments recursively </ins> # Pre-condition: no deployments exist kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" '' # Command

cc
Assigning to per test-
Flaked again
The flake actually happened when pausing the deployment. Changing issue title.
When , sometimes one of the deployments isn't paused. cc
I can't access the internal Google logs to examine the cause, nor can I reproduce this issue locally. Is there any additional information that you can provide to help troubleshoot the flake?
I've gone ahead and attempted a fix for this in
We should fix this in eventually
the external links are here ():


Question: errB := errors.New("binder") table := []struct { <del> name             string injectBindError  error sendPod          *v1.Pod algo             core.ScheduleAlgorithm expectErrorPod   *v1.Pod expectForgetPod  *v1.Pod expectAssumedPod *v1.Pod expectError      error expectBind       *v1.Binding eventReason      string </del> <ins> name                string injectBindError     error sendPod             *v1.Pod algo                core.ScheduleAlgorithm registerPluginFuncs []st.RegisterPluginFunc expectErrorPod      *v1.Pod expectForgetPod     *v1.Pod expectAssumedPod    *v1.Pod expectError         error expectBind          *v1.Binding eventReason         string </ins> }{ { <ins> name:    "error reserve pod", sendPod: podWithID("foo", ""), algo:    mockScheduler{core.ScheduleResult{SuggestedHost: testNode.Name, EvaluatedNodes: 1, FeasibleNodes: 1}, nil}, registerPluginFuncs: []st.RegisterPluginFunc{ st.RegisterReservePlugin("FakeReserve", st.NewFakeReservePlugin(framework.NewStatus(framework.Error, "reserve error"))), }, expectErrorPod:   podWithID("foo", testNode.Name), expectForgetPod:  podWithID("foo", testNode.Name), expectAssumedPod: podWithID("foo", testNode.Name), expectError:      errors.New(`error while running Reserve in "FakeReserve" reserve plugin for pod "foo": reserve error`), eventReason:      "FailedScheduling", }, { name:    "error permit pod", sendPod: podWithID("foo", ""), algo:    mockScheduler{core.ScheduleResult{SuggestedHost: testNode.Name, EvaluatedNodes: 1, FeasibleNodes: 1}, nil}, registerPluginFuncs: []st.RegisterPluginFunc{ st.RegisterPermitPlugin("FakePermit", st.NewFakePermitPlugin(framework.NewStatus(framework.Error, "permit error"), time.Minute)), }, expectErrorPod:   podWithID("foo", testNode.Name), expectForgetPod:  podWithID("foo", testNode.Name), expectAssumedPod: podWithID("foo", testNode.Name), expectError:      errors.New(`error while running "FakePermit" permit plugin for pod "foo": permit error`), eventReason:      "FailedScheduling", }, { name:    "error prebind pod", sendPod: podWithID("foo", ""), algo:    mockScheduler{core.ScheduleResult{SuggestedHost: testNode.Name, EvaluatedNodes: 1, FeasibleNodes: 1}, nil}, registerPluginFuncs: []st.RegisterPluginFunc{ st.RegisterPreBindPlugin("FakePreBind", st.NewFakePreBindPlugin(framework.NewStatus(framework.Error, "prebind error"))), }, expectErrorPod:   podWithID("foo", testNode.Name), expectForgetPod:  podWithID("foo", testNode.Name), expectAssumedPod: podWithID("foo", testNode.Name), expectError:      errors.New(`error while running "FakePreBind" prebind plugin for pod "foo": prebind error`), eventReason:      "FailedScheduling", }, { </ins> name:             "bind assumed pod scheduled", sendPod:          podWithID("foo", ""), algo:             mockScheduler{core.ScheduleResult{SuggestedHost: testNode.Name, EvaluatedNodes: 1, FeasibleNodes: 1}, nil},

This is a bug found in The scheduler fails to schedule the pod on the first try due to PV is out of sync. This may happen because the PV/PVC may be modified before the scheduler binder assumes them. However, the scheduler skips the pod update event and never try to scheduler the pod again: This is because when the reserve plugins fails, we didn't try to forget the pod. After the is called, the pod is in the state.
/sig scheduling /sig storage /assign /priority critical-urgent
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduling /sig storage /assign /priority critical-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // restarted. status = pod.Status } <del> if status.Phase == v1.PodFailed || status.Phase == v1.PodSucceeded { return true } return false </del> <ins> return status.Phase == v1.PodFailed || status.Phase == v1.PodSucceeded || (pod.DeletionTimestamp != nil && notRunning(status.ContainerStatuses)) </ins> } // OkToDeletePod returns true if all required node-level resources that a pod was consuming have

https://k8s- Only one green run out of the past 11 runs. Potential offenders: /cc (saw you guys discussed this on sig-node slack)
Related test failures: , ,
assigning this to you. and I can help out if necessary.
Still not sure what's going on. In 1 downward api modification failure, I saw a pod created, container started, container killed 1 second later by the kubelet, when it's supposed to run for 120s. Very strange.
See e.g.
So far I've only found the volume tests failing in the ubuntu trusty docker 1.10 vms. Not sure if that's 100% of them or just what I've looked at so far.
One idea is that cgroups are cleaned up before the pod is started, and that somehow causes the pod to fail. See . I'll try to verify my theory later...
That's consistent with my observation. Ubuntu+docker 1.10 seems to be the only one failing. If anyone wants to try reproducing this locally, this would be the image to use.
I noticed that too. The "pod ran to completion" is "pod started and immediately exited". I've tried to reproduce locally and can't find a commit around last Wednesday that obviously would have caused this. I'll post any information I find here.
I tried to reproduce locally (not on GCE) with trusty & docker 1.10 but the tests are all passing so far. Either I didn't wait long enough or it's something specific to the GCE trusty image being used.
-- i am not sure how that is possible. the odd bit to me is we ran w/ pod cgroups on in node e2e since 1.5 if i am not mistaken.
-- ignore my previous comment as previously we only deleted the cgroup when the pod was removed from the api server, and not just terminated. so yeah, its possible something is happening here...
maybe we need to pass in both activePods and runningPods to get a more accurate listing?
see from . Search for pod , which has cgroups cleaned up within 2 seconds of starting. May not be the culplit, but its worth looking into.

the runningPods cache appears like it could have a 2 second latency before updating its cache so it could be related and could be helped by passing in activePods as well...
see:
That is possible. I am also not sure when a pod is considered running, and in what order pods are created. Do we create cgroups before starting the container? Would a pod be returned by GetPods (from above comment) if it had cgroups set up, but the container wasn't running yet? It could also simply be that we are cleaning up a pod's cgroups before it is ever running (if that is possible based on above questions).
-- we create the cgroup before the container is started, so I think we need to pass activePods. see:


Question: --quiet  --delete-disks boot  --zone "${ZONE}"  <del> "${minions[@]::10}" || true </del> <ins> "${minions[@]::10}" </ins> minions=( "${minions[@]:10}" ) done

'kubernetes-soak-weekly-deploy' sometimes fails to deploy because the clean-up cluster step fails to delete the template: Then the subsequent cluster step fails to create the template, because it already exists: It keeps retrying and failing, because the template already exists, until it times out and the deployment fails. Suggested fixes: Cluster down should try harder to delete the template (maybe wait/retry). Cluster up should recognize that the template exists, and try to delete/recreate, or just re-use the existing.
Looks like ran in to this on Monday, May 11, 2015 around 1:00 PM PST. I'll assign this to my self and take a crack at fixing it.
This is causing most of our e2e tests to fail in continuous integration as well as users private e2e test clusters. Bumping to priority/p0 and v1.0 milestone. Saad thinks he's figured it out, and is working on a fix as we speak.
In essence... Saad: "I think what's happening is the previous clean up step to delete the managed instance group happens asynchronously, therefore there is a race condition to delete the template. Working on a fix".


Question: "!('k3' in self.val1)", "self.val1 == {'k1': 'a', 'k2': 'b'}", }, <ins> errors: map[string]string{ // Mixed type maps are not allowed since we have HomogeneousAggregateLiterals enabled "{'k1': 'a', 'k2': 1, 'k2': false}": "expected type 'string' but found 'int'", }, </ins> }, {name: "objects", obj: map[string]interface{}{

Added this test to : this resulted in: The test should pass. Add the above test No response All versions where CEL is used. All Doesn't matter No response No response No response_
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Opened to discuss the expected behavior in cel-go for this.


Question: Name:      "probe_result", Help:      "The result of a liveness or readiness probe for a container.", }, <del> []string{"probe_type", "container_name", "pod_name", "namespace", "pod_uid"}, </del> <ins> []string{"probe_type", "container_name", "container", "pod_name", "pod", "namespace", "pod_uid"}, </ins> ) // Manager manages pod probing. It creates a probe "worker" for every container that specifies a

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What's happening: Currently, the metrics use the label "podname" to refer to the name of the pod in a timeseries and "containername" to refer to container names within that pod. kube-state-metrics, however, uses "pod" to refer to the names of pods and "container" to refer to names of containers. This difference makes it difficult to do joins on KSM metrics and cadvisor metrics, leading me to write queries like this one (which calculates the percentage of real utilization of quota per container -- namespace and podname templated in Grafana): With normalized labels I could just write Hence, I propose renaming the "podname" labels to "pod" and "containername" labels to "container" to normalize between the two data sources. I can attempt to write a patch for this. Why propose this here rather than telling KSM to match the cadvisor metrics? I reported this against KSM first: The indicate the usage of "container" (vs. containername), "pod" (vs. podname), etc.: Fixing this would ensure the cadvisor metrics conform with the community guidelines. Environment: Kubernetes version (use ): 1.8.7-21 with internal patches Kube-state-metrics image version: 1.2.0
/sig instrumentation
I just this to be discussed in the sig-instrumentation meeting, personally I'm all for it and we've discussed this a number of times. we should figure out the timeline for node-metrics being reworked, if that is going to take a while then I think we should make these changes.
considering that now basically has a dependent change that has to go through first, I think that's ok
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: name:                      "reject redirection enabled in proxy, backend server sending 301 response", rejectForwardingRedirects: true, serverStatusCode:          301, <ins> redirect:                  "/", </ins> expectStatusCode:          502, expectBody:                []byte(`the backend attempted to redirect this request, which is not permitted`), }, { <ins> name:                      "reject redirection enabled in proxy, backend server sending 304 response with a location header", rejectForwardingRedirects: true, serverStatusCode:          304, redirect:                  "/", expectStatusCode:          502, expectBody:                []byte(`the backend attempted to redirect this request, which is not permitted`), }, { name:                      "reject redirection enabled in proxy, backend server sending 304 response with no location header", rejectForwardingRedirects: true, serverStatusCode:          304, expectStatusCode:          304, expectBody:                []byte{}, // client doesn't read the body for 304 responses }, { </ins> name:                      "reject redirection disabled in proxy, backend server sending 200 response", rejectForwardingRedirects: false, serverStatusCode:          200,

Upgraded from Kubernetes v1.25.0 to v1.25.1. Standard metrics-server deployment now causes the following message to appear in the kube-apiserver logs on a recurring basis: Adding a quick hack debug Infof print shows: It appears that the apiserver is sending an If-None-Match cache control header when refreshing the openapi spec, and the metrics-server service is correctly responding with a 304 Not Modified. No errors logged; proper refresh of OpenAPI specs Deploy metrics-server, or any other service that uses apiserver aggregation and honors caching headers such as , to a Kubernetes 1.25.1 cluster. made bad assumptions; not all HTTP 3XX responses are redirects. It should probably only reject the response if it is 300-399 AND the header is set. <details</detailsnone <details</detailsk3s containerd flannel
Taking a stab at some tags here... /cc /cc /cc /area apiserver /sig api-machinery
thanks for the report, looks like this should be /priority critical-urgent /kind regression cc since this requires a respin of a final 1.22 patch release :-/
Fix open in , will get cherry-picks open to start greening up CI
/triage accepted
I need to make a redirect to the external URL but it's not allowed :(


Question: return configFromSim() } <ins> func init() { klog.InitFlags(nil) } </ins> func TestSecretUpdated(t *testing.T) { datacenter := "0.0.0.0" secretName := "vccreds"

/kind failing-test /kind flake /sig cloud-provider /area provider/vmware Flaked on Is this the same issue as ?
/assign There seems to be a data-race here.
There's a finalizer that trips up the test case. TestSecretUpdated set flag and read from so when the logout read and the test is write the ,they lead to I came up with a mutex where the klog library reads Or a lock on the kubernetes vsphere logout function. Any better suggestions? /cc
the use of a runtime finalizer is suspect... that's the only use in our codebase and makes unit testing really hard. also, the use of the global logger in TestSecretUpdated is suspect... the code under test should be structured to allow testing the results without plumbing through the global logger
/cc
/assign
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Since the test is still failed, pick up at /cc


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Based on discussion , events can be throttled and should not be used to validate failure conditions in e2e tests. We should remove the event checks from subpath and local pv tests and just wait for pod failure. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
could you help look at this for local PV tests?
sure /assign
/area deflake
"Invalid AWS KMS key" also waits for an event

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup tests are flaking 1-2 times a day: https://k8s- Some test logs: https://k8s- https://k8s-
could you take a look?
I'll look into this.
I found why the tests failed, see my analysis of logs: I created a PR () trying to fix the issue. I am still investigating kubelet and trying to reproduce the issue locally.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen


Question: eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: kubeClient.CoreV1().Events("")}) cc := &CertificateController{ <ins> name:       name, </ins> kubeClient: kubeClient, queue: workqueue.NewNamedRateLimitingQueue(workqueue.NewMaxOfRateLimiter( workqueue.NewItemExponentialFailureRateLimiter(200*time.Millisecond, 1000*time.Second),

There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "sync" "testing" "time" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/fields" "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/util/wait" ) // verifies the cacheWatcher.process goroutine is properly cleaned up even if // the writes to cacheWatcher.result channel is blocked. func TestCacheWatcherCleanupNotBlockedByResult(t *testing.T) { var lock sync.RWMutex count := 0 filter := func(string, labels.Set, fields.Set) bool { return true } forget := func(bool) { lock.Lock() defer lock.Unlock() count++ } initEvents := []watchCacheEvent{ {Object: &api.Pod{}}, {Object: &api.Pod{}}, } // set the size of the buffer of w.result to 0, so that the writes to // w.result is blocked. w := newCacheWatcher(0, 0, initEvents, filter, forget) w.Stop() if err := wait.PollImmediate(1*time.Second, 5*time.Second, func() (bool, error) { lock.RLock() defer lock.RUnlock() return count == 2, nil }); err != nil { t.Fatalf("expected forget() to be called twice, because sendWatchCacheEvent should not be blocked by the result channel: %v", err) } } </ins>

We noticed many goroutines stalled at in a large cluster (~200 nodes, ~1100 pods). It's because if the WatchServer timeout when the channel is full, will be blocked on the write to the even though is , because WatchServer stops consuming the channel after the timeout.
Great finding - thanks for debugging


Question: service:         getTestService("test1", v1.ProtocolTCP, map[string]string{"service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset": "true"}, false, 80), loadBalancerSku: "basic", wantLb:          true, <del> expectedProbes: []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol("Tcp"), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, }, expectedRules: []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, EnableTCPReset: nil, }, }, }, </del> <ins> expectedProbes:  getDefaultTestProbes("Tcp", ""), expectedRules:   getDefaultTestRules(false), </ins> }, { desc:            "reconcileLoadBalancerRule shall return corresponding probe and lbRule (slb without tcp reset)", service:         getTestService("test1", v1.ProtocolTCP, map[string]string{"service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset": "True"}, false, 80), loadBalancerSku: "standard", wantLb:          true, <del> expectedProbes: []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol("Tcp"), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, }, expectedRules: []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, EnableTCPReset: to.BoolPtr(true), }, }, }, </del> <ins> expectedProbes:  getDefaultTestProbes("Tcp", ""), expectedRules:   getDefaultTestRules(true), </ins> }, { desc:            "reconcileLoadBalancerRule shall return corresponding probe and lbRule(slb with tcp reset)", service:         getTestService("test1", v1.ProtocolTCP, nil, false, 80), loadBalancerSku: "standard", wantLb:          true, <del> expectedProbes: []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol("Tcp"), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, }, expectedRules: []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, EnableTCPReset: to.BoolPtr(true), }, }, }, </del> <ins> expectedProbes:  getDefaultTestProbes("Tcp", ""), expectedRules:   getDefaultTestRules(true), }, { desc:            "reconcileLoadBalancerRule shall respect the probe protocol and path configuration in the config file", service:         getTestService("test1", v1.ProtocolTCP, nil, false, 80), loadBalancerSku: "standard", wantLb:          true, probeProtocol:   "http", probePath:       "/healthy", expectedProbes:  getDefaultTestProbes("http", "/healthy"), expectedRules:   getDefaultTestRules(true), </ins> }, } for i, test := range testCases { az := GetTestCloud(ctrl) az.Config.LoadBalancerSku = test.loadBalancerSku <ins> service := test.service if test.probeProtocol != "" { service.Annotations[ServiceAnnotationLoadBalancerHealthProbeProtocol] = test.probeProtocol } if test.probePath != "" { service.Annotations[ServiceAnnotationLoadBalancerHealthProbeRequestPath] = test.probePath } </ins> probe, lbrule, err := az.reconcileLoadBalancerRule(&test.service, test.wantLb, "frontendIPConfigID", "backendPoolID", "lbname", to.Int32Ptr(0))

What would you like to be : Support for additional load balancer health probe configurations (specifically HTTP and HTTPS) for Azure Load Balancers. Why is this needed: The generated Azure Load Balancers for type services only support TCP health probes. Those guarantee that the backend pool consists of nodes listening on the target port but do not validate whether or not anything is responding. We've also noticed elevated volumes SSL handshake errors generated by the TCP health checks targeting 443 when using more verbose log level configurations in ingress-nginx. It would be valuable to configure the type of health probe used by the AKS-created Azure Load Balancer.
/sig azure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any update on this?
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about this feature requests?
I had a look at a possible implementation. I did not create a PR yet, because the relevant files are in and the said there should only be bug fixes, no new features. In , there is some logic which would lead to the load balancer created with protocol HTTP (which is what we want) The check is from here: and the "RequestsOnlyLocalTraffic" is about the : However, I do not understand the relationship here, why setting should result in HTTP health check. What I think we could do is add a new annotation like and when this is set to , the health check will use the HTTP protocol in .

We could also add an additional annotation to set the property of the health check probe, this could be used so that people can set the path to or similar.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We have a similar requirement for having an http health probe with a custom path. According to comment, we could get an http probe by setting but our service expects health probes on a different path, not . Looks like proposed solution would work for us too. Are there any workarounds to achieve this currently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign