Question: <ins> // +build windows /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dockershim import ( "bytes" "fmt" "io" "k8s.io/kubernetes/pkg/kubelet/util/ioutils" ) func (r *streamingRuntime) portForward(podSandboxID string, port int32, stream io.ReadWriteCloser) error { stderr := new(bytes.Buffer) err := r.exec(podSandboxID, []string{"wincat.exe", "localhost", fmt.Sprint(port)}, stream, stream, ioutils.WriteCloserWrapper(stderr), false, nil, 0) if err != nil { return fmt.Errorf("%v: %s", err, stderr.String()) } return nil } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Ran on windows pod and received the error message: What you expected to happen: The port be available on my localhost How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: socat is not available on windows nodes Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server version: 1.10.2 Cloud provider or hardware configuration: Azure - acs-engine OS (e.g. from /etc/os-release): Windows Server 1803 Kernel (e.g. ): Install tools: Others:
/sig windows
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can confirm this on 1803:
What would be the approach to fixing this? If connecting to a windows machine use a different tool other than socat? Are there any tools that are used in the windows world for port-forwarding? If I got some guidance on tools to use I could take a try at a PR for this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The problem is that the portForward streaming operation invokes and on the node. Those aren't available (or required) on Windows.

On Linux, the command is something like
Would using be the command to get this to work? (v=ws.10)#add-v4tov4
/remove-lifecycle stale
I looked into this, I think that it could in theory work, the hard part about trying to use is that it needs the IP address, whereas in this method all you have is the PID and . I don't know how to do mapping to get the IP, but if you can figure that out then it's just a matter of allocating a random port to use as the listen port, assuming people are okay with that solution. It has the disadvantage of being less secure than the Linux side, which only exposes access to the stream to the Kubelet process instead of all processes, but I'm not sure that's a big security concern.
This function implemented by the CRI shim, which is capable of finding out the IP address by . /cc
I'm a bit ignorant of all things windows. Does "all processes" really mean everything, including unprivileged containers, or do you just mean all processes in the host network namespace (or whatever the windows equivalent is)?
It would have been just the host network namespace. This approach didn't really pan out though. is investigating another approach that more closely matches the way it works on Linux.


Question: // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) <del> if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn(podBackend2)); err != nil { </del> <ins> if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn(podBackend2, podClient)); err != nil { </ins> logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) framework.Failf("Failed to connect to backend 2") } }) <ins> // Regression test for #74839, where: // Packets considered INVALID by conntrack are now dropped. In particular, this fixes // a problem where spurious retransmits in a long-running TCP connection to a service // IP could result in the connection being closed with the error "Connection reset by // peer" // xref: https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/ ginkgo.It("should drop INVALID conntrack entries", func() { serverLabel := map[string]string{ "app": "boom-server", } serverPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "boom-server", Labels: serverLabel, }, Spec: v1.PodSpec{ NodeName: serverNodeInfo.name, Containers: []v1.Container{ { Name:  "boom-server", Image: imageutils.GetE2EImage(imageutils.RegressionIssue74839), Ports: []v1.ContainerPort{ { ContainerPort: 9000, // Default port exposed by boom-server }, }, Env: []v1.EnvVar{ { Name: "POD_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIP", }, }, }, { Name: "POD_IPS", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIPs", }, }, }, }, }, }, }, } _, err := fr.ClientSet.CoreV1().Pods(fr.Namespace.Name).Create(context.TODO(), serverPod, metav1.CreateOptions{}) framework.ExpectNoError(err) err = e2epod.WaitForPodsRunningReady(fr.ClientSet, fr.Namespace.Name, 1, 0, framework.PodReadyBeforeTimeout, map[string]string{}) framework.ExpectNoError(err) ginkgo.By("Server pod created on node " + serverNodeInfo.name) svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "boom-server", }, Spec: v1.ServiceSpec{ Selector: serverLabel, Ports: []v1.ServicePort{ { Protocol: v1.ProtocolTCP, Port:     9000, }, }, }, } _, err = fr.ClientSet.CoreV1().Services(fr.Namespace.Name).Create(context.TODO(), svc, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Server service created") pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "startup-script", }, Spec: v1.PodSpec{ NodeName: clientNodeInfo.name, Containers: []v1.Container{ { Name:  "startup-script", Image: imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{ "sh", "-c", "while true; do sleep 2; nc boom-server 9000& done", }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } _, err = fr.ClientSet.CoreV1().Pods(fr.Namespace.Name).Create(context.TODO(), pod, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Client pod created") // The client will open connections against the server // The server will inject invalid packets // if conntrack does not drop the invalid packets it will go through without NAT // so the client will receive an unexpected TCP connection and RST the connection // the server will log ERROR if that happens ginkgo.By("checking client pod does not RST the TCP connection because it receives and INVALID packet") if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn("ERROR", "boom-server")); err == nil { logs, err := e2epod.GetPodLogs(cs, ns, "boom-server", "boom-server") framework.ExpectNoError(err) framework.Logf("boom-server pod logs: %s", logs) framework.Failf("Boom server pod received a RST from the client") } logs, err := e2epod.GetPodLogs(cs, ns, "boom-server", "boom-server") framework.ExpectNoError(err) if !strings.Contains(string(logs), "connection established") { framework.Failf("Boom server pod did not sent any bad packet to the client") } framework.Logf("boom-server pod logs: %s", logs) framework.Logf("boom-server did not receive any RST packet") }) </ins> })

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }) } } <ins> func Test_csiMountMgr_supportsFSGroup(t *testing.T) { type fields struct { plugin              *csiPlugin driverName          csiDriverName volumeLifecycleMode storage.VolumeLifecycleMode fsGroupPolicy       storage.FSGroupPolicy volumeID            string specVolumeID        string readOnly            bool supportsSELinux     bool spec                *volume.Spec pod                 *api.Pod podUID              types.UID publishContext      map[string]string kubeVolHost         volume.KubeletVolumeHost MetricsProvider     volume.MetricsProvider } type args struct { fsType       string fsGroup      *int64 driverPolicy storage.FSGroupPolicy } tests := []struct { name   string fields fields args   args want   bool }{ { name: "empty all", args: args{}, want: false, }, { name: "driverPolicy is FileFSGroupPolicy", args: args{ fsGroup:      new(int64), driverPolicy: storage.FileFSGroupPolicy, }, want: true, }, { name: "driverPolicy is ReadWriteOnceWithFSTypeFSGroupPolicy", args: args{ fsGroup:      new(int64), driverPolicy: storage.ReadWriteOnceWithFSTypeFSGroupPolicy, }, want: false, }, { name: "driverPolicy is ReadWriteOnceWithFSTypeFSGroupPolicy with empty Spec", args: args{ fsGroup:      new(int64), fsType:       "ext4", driverPolicy: storage.ReadWriteOnceWithFSTypeFSGroupPolicy, }, fields: fields{ spec: &volume.Spec{}, }, want: false, }, { name: "driverPolicy is ReadWriteOnceWithFSTypeFSGroupPolicy with empty PersistentVolume", args: args{ fsGroup:      new(int64), fsType:       "ext4", driverPolicy: storage.ReadWriteOnceWithFSTypeFSGroupPolicy, }, fields: fields{ spec: volume.NewSpecFromPersistentVolume(&corev1.PersistentVolume{}, true), }, want: false, }, { name: "driverPolicy is ReadWriteOnceWithFSTypeFSGroupPolicy with empty AccessModes", args: args{ fsGroup:      new(int64), fsType:       "ext4", driverPolicy: storage.ReadWriteOnceWithFSTypeFSGroupPolicy, }, fields: fields{ spec: volume.NewSpecFromPersistentVolume(&api.PersistentVolume{ Spec: api.PersistentVolumeSpec{ AccessModes: []api.PersistentVolumeAccessMode{}, }, }, true), }, want: false, }, { name: "driverPolicy is ReadWriteOnceWithFSTypeFSGroupPolicy with ReadWriteOnce AccessModes", args: args{ fsGroup:      new(int64), fsType:       "ext4", driverPolicy: storage.ReadWriteOnceWithFSTypeFSGroupPolicy, }, fields: fields{ spec: volume.NewSpecFromPersistentVolume(&api.PersistentVolume{ Spec: api.PersistentVolumeSpec{ AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce}, }, }, true), }, want: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { c := &csiMountMgr{ plugin:              tt.fields.plugin, driverName:          tt.fields.driverName, volumeLifecycleMode: tt.fields.volumeLifecycleMode, fsGroupPolicy:       tt.fields.fsGroupPolicy, volumeID:            tt.fields.volumeID, specVolumeID:        tt.fields.specVolumeID, readOnly:            tt.fields.readOnly, supportsSELinux:     tt.fields.supportsSELinux, spec:                tt.fields.spec, pod:                 tt.fields.pod, podUID:              tt.fields.podUID, publishContext:      tt.fields.publishContext, kubeVolHost:         tt.fields.kubeVolHost, MetricsProvider:     tt.fields.MetricsProvider, } if got := c.supportsFSGroup(tt.args.fsType, tt.args.fsGroup, tt.args.driverPolicy); got != tt.want { t.Errorf("supportsFSGroup() = %v, want %v", got, tt.want) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet panics on creation of inline ephemeral volume with FSType specified. creation of inline ephemeral volume succeeds with FSType specified. no existing isolated test case. Happens during e2e testing of pmem-csi. Kubernetes version (use ): The pmem-csi CI system where this was detected, runs same tests on k8s systems with 1.19 and 1.20. The problem occurs in testing on 1.20 system, problem does not occur on 1.19 system.Cloud provider or hardware configuration: pmem-csi tests run on cluster formed by few qemu-based systems running Fedora-33 and emulated pmem storage. The issue has been seen in e2e testing in , where a new test was recently introduced that creates a pod with an inline ephemeral volume. If the related VolumeSource has filesystem type set to empty "", or FSType is omitted in spec, then the pod creation works OK. In that case, pmem-csi chooses it's default fstype which is . Example of working definition of VolumeSource: If the filesystem type in the VolumeSource is set to or then kubelet fails to create the volume and pod. A panic trace is written in the node kubelet log and the creation will be retried few times, On pmem-csi side: the 1st NodePublishVolume is received, namespace and volume are created, formatted with mkfs, mounted and NodePublishVolume returns OK. Same request is received few times with increasing time interval. e2e testing code which tries to create the pod, will fail with:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node storage
... and ownership changed for those volumes.


Question: imageService internalapi.ImageManagerService // hostStatsProvider is used to get the status of the host filesystem consumed by pods. hostStatsProvider HostStatsProvider <ins> //lint:ignore U1000 We can't import hcsshim due to Build constraints in hcsshim // windowsNetworkStatsProvider is used by kubelet to gather networking stats on Windows windowsNetworkStatsProvider interface{} // clock is used report current time clock clock.Clock </ins> // cpuUsageCache caches the cpu usage for containers. cpuUsageCache                  map[string]*cpuUsageRecord

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When curling the endpoint for stats we don't see network stats. Network stats to be included when calling the summary. This works with dockershim for windows. This happens because Windows containerd uses hcsv2 to create containers and we query for the networks stats using hcsv1: This will be address eventually with move to CRI only metrics KEP: but is still several releases away. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: // - test/e2e_node/conformance/run_test.sh. "--pod-cidr", "10.100.0.0/24", "--eviction-pressure-transition-period", "30s", <del> // Apply test framework feature gates by default. This could also be overridden // by kubelet-flags. "--feature-gates", framework.TestContext.FeatureGates, </del> "--eviction-hard", "memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%", // The hard eviction thresholds. "--eviction-minimum-reclaim", "nodefs.available=5%,nodefs.inodesFree=5%", // The minimum reclaimed resources after eviction. "--v", LOG_VERBOSITY_LEVEL, "--logtostderr", ) <ins> // Apply test framework feature gates by default. This could also be overridden // by kubelet-flags. if framework.TestContext.FeatureGates != "" { cmdArgs = append(cmdArgs, "--feature-gates", framework.TestContext.FeatureGates) } </ins> if utilfeature.DefaultFeatureGate.Enabled(features.DynamicKubeletConfig) { // Enable dynamic config if the feature gate is enabled dynamicConfigDir, err := getDynamicConfigDir()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I want to run test on (ubuntu) with this cmd: What you expected to happen: e2enode test pass How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: After go into details I got this info: I think e2enode start kubelet by this cmd as is not set by default in : After fix it temporarily I got one more error about this issue: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: I will send a PR to fix this if it is confirmed.
/sig testing
/sig node
If your environment doesn't support swap, you could override kubelet flag with:
Thanks! Could you take a look at this : append if needed.


Question: return } <ins> // its safe to set initialSync to false as it acts as a flag for startup actions // and the mutex is held. defer func() { proxier.initialSync = false }() </ins> // Keep track of how long syncs take. start := time.Now() defer func() {

Entry in ipvs real sever remains at weight=0 after kube-proxy restarts. (During 1.21 -1.22 kubernetes cluster upgrade.) Related to As the related issue is considered closed, I will report this issue again as the problem still persists in newer version. Might because of some corner cases. Real server weight should be 1 when endpoints is ready. Use kube-proxy in ipvs mode Deploy a service resource and its backend pods Make real sever not ready (or may need node not ready) so as to trigger graceful delete in kube-proxy Before kube-proxy deletes the entry from ipvs, restart kube-proxy Kube-proxy becomes ready and leave the weight=0 entry as it is Service is no longer accessible through cluster ip When accessed, dmesg shows kernel messge error ofDeletion of pod will trigger kube-proxy to remove weight=0 entry from ipvs so that after the entry is recreated service will become accessible If service is 1. LoadBalancer, and, 2. .spec.externalTrafficPolicy: Local, and, 3, service is accessed from loadbalancer IP, the entry in ipvs for loadbalancer IP will only have endpoints in the same node. If backends is DaemonSet, the only one endpoints will have weight=0 thus causing connection failure After kube-proxy restarts, endpoints will be processed in the following lines. Only when the entry is in the gracefuldeleteManager.InTerminationList will be deleted. Which is not the case if kube-proxy restarts before it deletes the ipvs entry. An existing entry in the ipvs is sufficient for endpoint to be considered healthy. Weight is not considered. kube-proxy should treat ipvs entry properly by the either the following two ways. kube-proxy should initiate InTerminationList and put ipvs entry with weight=0 into the termination list kube-proxy should also look for weight in the ipvs entry instead of only UniqueRSName <details</details<detailson-premises </details<details</details<details</details<details</details<details</details
/sig network
Graceful delete of ipvs is not only triggered when pod is in terminating phase. Graceful delete seems to be triggered when endpointslice status is ready=false. This will happen when node become not ready.

However, after the node become ready again, pods on the node will only be recreated if the duration of NotReady is longer than certain time. Otherwise the pod will just become ready again. If the node will not recreate the pods && kube-proxy set the pods to the graceful delete queue && kube-proxy restarts before deletion and all cache is cleared, kube-proxy will keep ipvs entry with weight=0 even if the real server is ready. Please correct me if I misunderstand something.
/area ipvs /area kube-proxy
Seems like a legit bug to me. I can't triage until wednesday though.
I can't recreate this when I try to follow thre instructions with pod not-ready rather than node not ready. When the pod is made ready again is not stated, so I tried; a connection to a pod it not-ready. The weight is set to 0, the connection prevails -start kube-proxy. The weight is still 0, connection still working the pod ready weight is set to 1 after a while, as it should
In the instruction is different, the pod is made ready while is away. That instruction shows the fault. The reproduce instruction in this issue is incorrect.
I triage this issue as a bug, but it's really a duplicate of which was never fixed (it seems). Don't know if the proper action is to close this issue and re-open . /triage accepted
Please instruct if this issue shall be closed as duplicate and re-opened PTAL at
Thank you for your triage Monitoring IPVS weights is costly, but needed while this problem persists. Any news? Adding the following lines should fix the problem but will also add loads.
That does seem costly to do on every endpoint sync. Is there an alternative solution we can explore where we assess all endpoints once when kube-proxy initially starts up?
This is a startup-only problem. It should be solved with a one-time action.
How about modifying to be ? We could then check and update the real server weights only when is true. Only and will be modified to call , all other users will call the function with false. Since these two functions, are only called during startup, we won't be doing this monitoring every endpoint sync.
cc:


Question: package config import ( <ins> "fmt" </ins> "io" "path" "strconv"

The '-context' does not seem to add any value
add priority and team please.
Blocked on
/sig cli


Question: podSpec.SecurityContext.FSGroupChangePolicy = podConfig.PodFSGroupChangePolicy } <ins> setVolumes(podSpec, podConfig.PVCs, podConfig.InlineVolumeSources, podConfig.PVCsReadOnly) SetNodeSelection(podSpec, podConfig.NodeSelection) return podSpec } func setVolumes(podSpec *v1.PodSpec, pvcs []*v1.PersistentVolumeClaim, inlineVolumeSources []*v1.VolumeSource, pvcsReadOnly bool) { </ins> var volumeMounts = make([]v1.VolumeMount, 0) var volumeDevices = make([]v1.VolumeDevice, 0) <del> var volumes = make([]v1.Volume, len(podConfig.PVCs)+len(podConfig.InlineVolumeSources)) </del> <ins> var volumes = make([]v1.Volume, len(pvcs)+len(inlineVolumeSources)) </ins> volumeIndex := 0 <del> for _, pvclaim := range podConfig.PVCs { </del> <ins> for _, pvclaim := range pvcs { </ins> volumename := fmt.Sprintf("volume%v", volumeIndex+1) if pvclaim.Spec.VolumeMode != nil && *pvclaim.Spec.VolumeMode == v1.PersistentVolumeBlock { volumeDevices = append(volumeDevices, v1.VolumeDevice{Name: volumename, DevicePath: "/mnt/" + volumename}) } else { volumeMounts = append(volumeMounts, v1.VolumeMount{Name: volumename, MountPath: "/mnt/" + volumename}) } <del> volumes[volumeIndex] = v1.Volume{Name: volumename, VolumeSource: v1.VolumeSource{PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ClaimName: pvclaim.Name, ReadOnly: podConfig.PVCsReadOnly}}} </del> <ins> volumes[volumeIndex] = v1.Volume{ Name: volumename, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: pvclaim.Name, ReadOnly:  pvcsReadOnly, }, }, } </ins> volumeIndex++ } <del> for _, src := range podConfig.InlineVolumeSources { </del> <ins> for _, src := range inlineVolumeSources { </ins> volumename := fmt.Sprintf("volume%v", volumeIndex+1) // In-line volumes can be only filesystem, not block. volumeMounts = append(volumeMounts, v1.VolumeMount{Name: volumename, MountPath: "/mnt/" + volumename}) volumes[volumeIndex] = v1.Volume{Name: volumename, VolumeSource: *src} volumeIndex++ } podSpec.Containers[0].VolumeMounts = volumeMounts podSpec.Containers[0].VolumeDevices = volumeDevices podSpec.Volumes = volumes <del> SetNodeSelection(podSpec, podConfig.NodeSelection) return podSpec </del> }

Apparently volumes with are incorrectly to : /sig storage /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: pip, err := az.reconcilePublicIP("testCluster", &service, "", test.wantLb) if test.expectedID != "" { assert.Equal(t, test.expectedID, to.String(pip.ID), "TestCase[%d]: %s", i, test.desc) <del> } else { assert.Equal(t, test.expectedPIP, pip, "TestCase[%d]: %s", i, test.desc) </del> <ins> } else if test.expectedPIP != nil && test.expectedPIP.Name != nil { assert.Equal(t, *test.expectedPIP.Name, *pip.Name, "TestCase[%d]: %s", i, test.desc) </ins> } assert.Equal(t, test.expectedError, err != nil, "TestCase[%d]: %s", i, test.desc) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: package csi import ( <ins> "context" </ins> "errors" "fmt" "os"

What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary


Question: switch pattern.VolType { case DynamicPV: r.Pv, r.Pvc = createPVCPVFromDynamicProvisionSC( <del> f, dInfo.Name, claimSize, r.Sc, pattern.VolMode, dInfo.RequiredAccessModes) </del> <ins> f, dInfo.Name, claimSize, r.Sc, pattern.VolMode, accessModes) </ins> r.VolSource = storageutils.CreateVolumeSource(r.Pvc.Name, false /* readOnly */) case GenericEphemeralVolume: driverVolumeSizeRange := dDriver.GetDriverInfo().SupportedSizeRange claimSize, err := storageutils.GetSizeRangesIntersection(testVolumeSizeRange, driverVolumeSizeRange) framework.ExpectNoError(err, "determine intersection of test size range %+v and driver size range %+v", testVolumeSizeRange, driverVolumeSizeRange) <del> r.VolSource = createEphemeralVolumeSource(r.Sc.Name, pattern.VolMode, dInfo.RequiredAccessModes, claimSize) </del> <ins> r.VolSource = createEphemeralVolumeSource(r.Sc.Name, pattern.VolMode, accessModes, claimSize) </ins> } } case CSIInlineVolume:

For the ReadWriteOncePod feature to graduate to beta, we need to have E2E test coverage for its current alpha behavior. See this section of KEP-2485 for more context on what cases we need to cover for alpha: This is a requirement for beta graduation, see:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: host, port, _ := net.SplitHostPort(listener.Addr().String()) t.Logf("Asked a %s forward for: %s:%v, got listener %s:%s, expected: %s", testCase.Protocol, testCase.Hostname, 12345, host, port, expectedListenerPort) if host != testCase.ExpectedListenerAddress { <del> t.Errorf("Test case #%d failed: Listener does not listen on exepected address: asked %v got %v", i, testCase.ExpectedListenerAddress, host) </del> <ins> t.Errorf("Test case #%d failed: Listener does not listen on expected address: asked '%v' got '%v'", i, testCase.ExpectedListenerAddress, host) </ins> } if port != expectedListenerPort { t.Errorf("Test case #%d failed: Listener does not listen on exepected port: asked %v got %v", i, expectedListenerPort, port)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When using IPv6 addresses, the kubelet passes the address to client-go, which does not correctly format the address for use in API requests. For example: Failed to list v1.Pod: Get https://2001:129::136:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dnode136&resourceVersi on=0: dial tcp: too many colons in address 2001:129::136:6443 What you expected to happen: The URI should have https://[2001:129::136]:6443, so that SplitHostPort() will correctly be able to separate the host and port components. How to reproduce it (as minimally and precisely as possible): Create cluster using IPv6 address (e.g. kubeadm init --apiserver-advertise-address=2001:129::136). When the kubelet starts up, it'll log this error. Anything else we need to know?: This is being handled as part of effort to support IPv6 in Kubernetes. Was found by a co-worker, I'll work on a commit to fix. Environment: Kubernetes version (use ): Using latest, via KubeAdm Cloud provider or hardware configuration*: bare-metal OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-81-generic Install tools: Using KubeAdm Others:
/area client-libraries /area kubelet /area ipv6
/assign
The point of failure is here: SplitHostPort will accept IPv4 and IPv6 addresses, but the latter must have square brackets around the IPv6 address.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery because client-go
Should we open this on the repository?

I see in the client-go repo read me that it states: Does that confirm that I should do a PR here?
One thing that I was wonder, was whether or not the hostname/address provided in the call to getListener, has been validated. If so, and it can only be a hostname, IPv4, or IPv6 address as a string, we could just check if there is a ":" in the name to determine IPv6. Otherwise, I could use net.ParseIP(), and net.To4() to determine if IPv6. I'll start working with the latter assumption, but please let me know, if the former could be used.
Also, I'm making the assumption that we'd want to make client-go be able to handle IPv6 addresses, rather than having each user of client-go (e.g. kubelet) from ensuring that the IP address is of the right form. IOW, have client-go add the needed [] around IPv6 addresses, rather than have kubelet do that. My thought is that there may be cases where we want just the IP address and not the modified IP. Does that sound OK?


Question: // NewIntegrationTestKubelet creates a new Kubelet for use in integration tests. // TODO: add more integration tests, and expand parameter list as needed. <del> func NewIntegrationTestKubelet(hn string, dc dockertools.DockerInterface) *Kubelet { </del> <ins> func NewIntegrationTestKubelet(hn string, rd string, dc dockertools.DockerInterface) *Kubelet { </ins> return &Kubelet{ hostname:              hn, dockerClient:          dc, <ins> rootDirectory:         rd, </ins> dockerPuller:          &dockertools.FakeDockerPuller{}, networkContainerImage: NetworkContainerImage, resyncInterval:        3 * time.Second,

E1003 10:53:11. ] Could not read directory: , (open : no such file or directory) This happens many times


Question: if len(ctr.Image) == 0 { allErrs = append(allErrs, field.Required(fldPath.Index(i).Child("image"), "")) } <ins> // this is only called from ValidatePodUpdate so its safe to check leading/trailing whitespace. if len(strings.TrimSpace(ctr.Image)) != len(ctr.Image) { allErrs = append(allErrs, field.Invalid(fldPath.Index(i).Child("image"), ctr.Image, "must not have leading or trailing whitespace")) } </ins> } return allErrs, false }

Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version (use ): kubernetes 1.5+ Environment: n/a What happened: examining events that happened in my cluster, i see an event like the following: this surprised me, as it implies a container had an imageName=" " What you expected to happen: i expected a validation failure error if a [i].imageName had leading or trailing whitespace. How to reproduce it (as minimally and precisely as possible): kubectl create -f my-pod kubectl edit pod/my-pod -o json change image to " " save Anything else we need to know: nope
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
discovered when looking at odd event messages in production cluster.


Question: } return false } <ins> func isStreamReset(err error) bool { if err == nil { return false } if urlErr, ok := err.(*url.Error); ok { // Sadly, the client does not receive a more specific indication // of stream reset. return strings.Contains(urlErr.Err.Error(), "INTERNAL_ERROR") } return false } </ins>

pull-kubernetes-unit TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter 7/14 No response /sig api-machinery
/priority important-soon
/cc /triage accepted
and can this be related to ?
I see no direct relationship, but hey, anything's possible. We learned that with the misery in integration test failures when we merged not-really-guilty other stuff.
The expanded logging turned up some new evidence. See , where the complaint is now
The problem here is not APF, it is something in the timeout filter or the test server.
is something in APF triggering a timeout?
The flake definitely came in with on the prior commit on master: on the merge commit of
/assign
Note that the flaking test is deliberately suffering server-side timeouts. The symptom in flakes is that the test is checking that the client gets kube's designed response to a timeout while handling, a particular form HTTP response message, but sometimes the client gets something else (an http2 internal error?). is a change in metrics, and should have nothing to do with how timeouts while handling and detected or handled. I will see what I can see...
, regarding your comment above (): note that the output shown (the panic "as designed") is normal; if there are failures involved, the failing part was elided.
Also, the failures seem to be coming from one particular subtest () and that one does not involve designed panics.
On closer examination of the timeout filter, it appears that this "INTERNAL ERROR" is something that can happen normally in the course of handling a timeout. So I will adjust the test to accept this.
I have updated to include the fix for this flake.


Question: "time" batch "k8s.io/api/batch/v1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels"

If the user update , duplicate condition: ! Only one Failed condition Create Job and wait it marked as Failed Update the Failed condition to False No response <details</details<details</details<details</details<details</details<details</details<details</details
It's not a bug. The is a list, so it can contain many conditions with different probe timestamps. Also, we should not update the fields.
But should we avoid the duplicate Type's conditionIn my opinionthis will confuse the user. According to kubelet, it will rewrite the 'Ready' condition in pod, and take is as managed only by kubelet. Should we did this for job Failed condition? Refer:
/sig apps
It is a bug. We should only have one condition for each type. /triage accepted
/assign
/assign
/assign
I have reproduced the issue locally and intend work on the fix
/priority backlog Since the user is manually modifying the status, this is not an inherent bug in the controller. Still worth avoiding duplicates.


Question: }), ) for arch in archs] for os, archs in SERVER_PLATFORMS.items()] <del> # The mondo test tarball is deprecated. pkg_tar( name = "_test-mondo-bin", srcs = ["//build:test-targets"], mode = "0755", package_dir = select(for_platforms( for_test = "platforms/{OS}/{ARCH}", )), tags = [ "manual", "no-cache", ], visibility = ["//visibility:private"], ) genrule( name = "kubernetes-test-mondo-deprecation", outs = ["DEPRECATION_NOTICE"], cmd = """cat <<EOF >$@ The mondo test tarball containing binaries for all platforms is DEPRECATED as of Kubernetes 1.14. Users of this tarball should migrate to using the platform-specific tarballs in combination with the "portable" tarball which contains scripts, test images, and other manifests. For more details, please see KEP sig-testing/20190118-breaking-apart-the-kubernetes-test-tarball. EOF """, visibility = ["//visibility:private"], ) # The mondo test tarball is deprecated. # This one was never really correct, anyway, since we can't include # binaries from multiple platforms in a single tarball. pkg_tar( name = "kubernetes-test", srcs = [ ":kubernetes-test-mondo-deprecation", "//build:test-portable-targets", ], extension = "tar.gz", package_dir = "kubernetes", remap_paths = { "build/release-tars/DEPRECATION_NOTICE": "DEPRECATION_NOTICE", }, strip_prefix = "//", tags = [ "manual", "no-cache", ], deps = select(for_platforms( for_test = [":_test-mondo-bin"], )), ) </del> pkg_tar( name = "kubernetes-test-portable", srcs = ["//build:test-portable-targets"],

What happened: On 2019-02-22 was merged with an indication that the mondo test tarball noted as deprecated as per But it did not include a timeline for eventual removal. What you expected to happen: It's been 7 months now. We should formalize a plan for removal. /sig testing /sig release /kind cleanup /priority important-soon
Enhancement tracking issue ref:
removes the deprecated mondo test tarball.


Question: } } if len(nper.To) == 0 { <del> w.Write(LEVEL_0, "%s%sn", initialIndent, "To: <any> (traffic not restricted by source)") </del> <ins> w.Write(LEVEL_0, "%s%sn", initialIndent, "To: <any> (traffic not restricted by destination)") </ins> } else { for _, to := range nper.To { w.Write(LEVEL_0, "%s%sn", initialIndent, "To:")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: A NetworkPolicy to allow egress to all Pods should use the term "destination" instead of "source" while example: What you expected to happen: should instead be kubectl describe networkpolicy net-document Name: net-document Namespace: default Created on: 2020-07-15 18:16:46 -0700 PDT Labels: <noneAnnotations: {"apiVersion":"","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"net-document","namespace":"default"},"spe... Spec: PodSelector: app=test-document Allowing ingress traffic: To Port: <any(traffic allowed to all ports) From: <any(traffic not restricted by source) Allowing egress traffic: To Port: 80/TCP To: <any(traffic not restricted by destination) Policy Types: Egress, Ingress `` How to reproduce it (as minimally and precisely as possible): Create a allow-all egress NetworkPolicy and do
/sig network
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/sig cli As this is something specific from kubectl :)
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cli Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/remove-triage unresolved


Question: <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") go_library( name = "go_default_library", srcs = ["logs.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/kuberuntime/logs", visibility = ["//visibility:public"], deps = [ "//pkg/kubelet/apis/cri:go_default_library", "//pkg/kubelet/apis/cri/v1alpha1/runtime:go_default_library", "//pkg/util/tail:go_default_library", "//vendor/github.com/docker/docker/pkg/jsonlog:go_default_library", "//vendor/github.com/fsnotify/fsnotify:go_default_library", "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", ], ) go_test( name = "go_default_test", srcs = ["logs_test.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/kuberuntime/logs", library = ":go_default_library", deps = [ "//vendor/github.com/stretchr/testify/assert:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], visibility = ["//visibility:public"], ) </ins>

We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok


Question: return list } <del> // Contains returns a util.StringSet containing all IDs of stored the items. </del> <ins> // ContainedIDs returns a util.StringSet containing all IDs of stored the items. </ins> // This is a snapshot of a moment in time, and one should keep in mind that // other go routines can add or remove items after you call this. <del> func (c *cache) Contains() util.StringSet { </del> <ins> func (c *cache) ContainedIDs() util.StringSet { </ins> c.lock.RLock() defer c.lock.RUnlock() set := util.StringSet{}

The client/cache interface has a method which is supposed to return a of the IDs in a store. This is a bit misleading and should probably be renamed to 'Contents' or 'ContainedIDs'.
Thoughts
I tend to agree. contains to me should always return a bool and take an input key. Sent from my iPhone
Yeah, I agree, both of the suggested names would be an improvement.


Question: name: new-kms-provider-for-configmaps cachesize: 1000 endpoint: unix:///@new-kms-provider.sock <del> - resources: - '*.*' providers: - kms: name: kms-provider-to-encrypt-all cachesize: 1000 endpoint: unix:///@new-encrypt-all-kms-provider.sock - identity: {} </del> ` <del> // start new KMS Plugin _ = mock.NewBase64Plugin(t, "@new-encrypt-all-kms-provider.sock") </del> // update encryption config and wait for hot reload <del> if err := os.WriteFile(filepath.Join(test.configDir, encryptionConfigFileName), []byte(encryptionConfigWithoutOldProvider), 0644); err != nil { t.Fatalf("failed to update encryption config, err: %v", err) } wantPrefixForEncryptAll := "k8s:enc:kms:v1:kms-provider-to-encrypt-all:" </del> <ins> updateFile(t, test.configDir, encryptionConfigFileName, []byte(encryptionConfigWithoutOldProvider)) </ins> // wait for config to be observed <del> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, wantPrefixForEncryptAll, test) </del> <ins> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, test) </ins> // confirm that reading secrets still works _, err = test.restClient.CoreV1().Secrets(testNamespace).Get(

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <del> // Mount : mounts source to target as NTFS with given options. </del> <ins> // Mount : mounts source to target with given options. // currently only supports cifs(smb), bind mount(for disk) </ins> func (mounter *Mounter) Mount(source string, target string, fstype string, options []string) error { target = normalizeWindowsPath(target) if source == "tmpfs" { <del> klog.V(3).Infof("azureMount: mounting source (%q), target (%q), with options (%q)", source, target, options) </del> <ins> klog.V(3).Infof("mounting source (%q), target (%q), with options (%q)", source, target, options) </ins> return os.MkdirAll(target, 0755) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.


Question: return "", err } <del> if !podEligibleToPreemptOthers(pod, fh.SnapshotSharedLister().NodeInfos()) { </del> <ins> if !podEligibleToPreemptOthers(pod, fh.SnapshotSharedLister().NodeInfos(), m[pod.Status.NominatedNodeName]) { </ins> klog.V(5).Infof("Pod %v/%v is not eligible for more preemption.", pod.Namespace, pod.Name) return "", nil }

What happened: When a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage, the scheduler doesn't consider that pod for preemption even there are pods can be preempted on another node. What you expected to happen: When the pod's nominated node is considered as 'UnschedulableAndUnresolvable ' (For example, the taint being by those conditions, ), the scheduler should try to preemption on other nodes. How to reproduce it (as minimally and precisely as possible): Create low priority pod, which schedule to nodeA Create mid priority pod, which schedule to nodeB Create high priority pod, which preempt the low priority pod on nodeA, and set high priority pod's nominated node as nodeA The nodeA's kubelet is down, the node life cycle controller taint the nodeA with '' (or taint with disk ) before low priority pod finishes the termination. The high priority pod is keeping as pending status. The scheduler doesn't try to preempt the mid-priority pod on nodeB Anything else we need to know?: Environment: Kubernetes version (use ): 18.0 Cloud provider or hardware configuration: baremetal OS (e.g: ): ubuntu1804 Kernel (e.g. ): 0-46-generic Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
/assign
This seems to be a duplicate of .
actually, I found this case after review . I file this issue since I feel it should be another case. The want to cleanup the nomnated node for cluster autoscaler. This is case need to allow the pod to preempt on other nodes again even it has a nomanted node. But I may understand incorrectly, please mark it duplicated if I'm wrong. Thanks for help anyway. :)
If the nominatedNodeName can be cleared properly in any case, this issue will no longer exist. From this point of view, I'd say this issue is a sub-case of .
BTW: in your description, if "The nodeA is down", how it comes out "a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage"? nodeA should NOT have existed in the result of Filter(), right?

(if the nodeA is just being shutdown, we still treat it as a normal node)
If the NodeA is down, the node life cycle controller will add taint '' to the node, then the TaintToleration filter will return 'UnschedulableAndUnresolvable '. It also means that it isn't only about nodeA is down, there are other conditions in this ' to return 'UnschedulableAndUnresolvable'. Beyond the TaintToleration filter, other filters may return 'UnschedulableAndUnresolvable' also, as my understanding, that means that node won't be available for that pod anymore, then we should try other nodes for preemption.
then we should better describe it as "being applied with a Taint", which is more general. When you say "node is down", it usually refers to the node is deleted physically.
I got it, thanks again! let me correct it
Also it's different from . Let's not mention "node is down" in the description to avoid confusion.
thanks, done :)


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/util/net" </ins> clientsetfake "k8s.io/client-go/kubernetes/fake" clienttesting "k8s.io/client-go/testing" extenderv1 "k8s.io/kube-scheduler/extender/v1"

If there is a connection failure when updating the Pod status, we don't retry. Furthermore, due to , we won't have another chance until much later. The problem is that if we don't mark the Pod as Unschedulable, other controllers (such as cluster-autoscaler) wouldn't react to these pods. We should have stronger guarantees to mark a Pod as Unschedulable. You need an unschedulable Pod (for example, a Pod with a non matching node affinity) and a flaky connection to apiserver. No response master This is particularly problematic in 1.24, due to . Any <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
That is bad, retrying in place on transient errors should probably be ok.
I'll address it. /assign
Thanks!


Question: return unmounter, nil } <ins> // newVolumeDetacherFromPlugins attempts to find a plugin by a name and then // create a Detacher. // Returns: //  - a detacher if one exists //  - an error if no plugin was found for the volume //    or the detacher failed to instantiate //  - nil if there is no appropriate detacher for this volume </ins> func (kl *Kubelet) newVolumeDetacherFromPlugins(kind string, name string, podUID types.UID) (volume.Detacher, error) { plugName := strings.UnescapeQualifiedNameForDisk(kind) plugin, err := kl.volumePluginMgr.FindAttachablePluginByName(plugName)

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: // RecordNodeEvent records a event related to a node. func RecordNodeEvent(recorder record.EventRecorder, nodeName, nodeUID, eventtype, reason, event string) { ref := &v1.ObjectReference{ <del> Kind:      "Node", Name:      nodeName, UID:       types.UID(nodeUID), Namespace: "", </del> <ins> APIVersion: "v1", Kind:       "Node", Name:       nodeName, UID:        types.UID(nodeUID), Namespace:  "", </ins> } klog.V(2).Infof("Recording %s event message for node %s", event, nodeName) recorder.Eventf(ref, eventtype, reason, "Node %s event: %s", nodeName, event)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm attempting to use an watcher to better re-act to cluster modifications. I've received the below event: What you expected to happen: I would expect the to include attribute. Not sure if it's all node events or just this particular one. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: bare metal, rancher's rke OS (e.g: ): centos host nodes Kernel (e.g. ): 3.18, 4.4 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
Just got another from kubelet..
/remove-sig api-machinery /sig node
recording code is here:


Question: /* Testname: Service, update NodePort, same port different protocol <del> Description: Create a service of type ClusterIP to accept TCP requests. Service creation MUST be successful by assigning ClusterIP to the service. When service type is updated to NodePort to support two protocols i.e. TCP and UDP for same assigned service port 80, service update MUST be successful by allocating two NodePorts to the service. TODO: Test Service reachability, good to include this check in Conformance perspective. </del> <ins> Description: Create a service to accept TCP requests. By default, created service MUST be of type ClusterIP and an ClusterIP MUST be assigned to the service. When service type is updated to NodePort supporting TCP protocol, it MUST be reachable on nodeIP over allocated NodePort to serve TCP requests. When this NodePort service is updated to use two protocols i.e. TCP and UDP for same assigned service port 80, service update MUST be successful by allocating two NodePorts to the service and service MUST be able to serve both TCP and UDP requests over same service port 80. </ins> */ <del> ginkgo.It("should be able to update NodePorts with two same port numbers but different protocols", func() { </del> <ins> ginkgo.It("should be able to update service type to NodePort listening on same port number but different protocols", func() { </ins> serviceName := "nodeport-update-service" ns := f.Namespace.Name jig := e2eservice.NewTestJig(cs, serviceName)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Following are two e2e testing service functioning over same port but different protocol, but do not test service reachability. be able to update NodePorts with two same port numbers but different protocols use same NodePort with same port but different protocols Note: Both the e2e are verifying the same behavior of "service using same port but different protocols" differs in and . Primary goal: Test service reachability for these E2Es. Secondary goal: As both the above e2e are verifying same E2E behavior, it's good to have single e2e in spite of 2 differing in "updating" and "create only" nodePort service. For "create only", suite has another specific e2e . Why is this needed: E2E Improvement and make e2e eligible for Conformance. Service related e2e are getting promoted in /area conformance /sig testing /assign


Question: } defer a.queue.Done(key) <del> err := a.reconcileKey(key.(string)) if err == nil { // don't "forget" here because we want to only process a given HPA once per resync interval return true </del> <ins> deleted, err := a.reconcileKey(key.(string)) if err != nil { utilruntime.HandleError(err) } // Add request processing HPA after resync interval just in case last resync didn't insert // request into the queue. Request is not inserted into queue by resync if previous one wasn't processed yet. // This happens quite often because requests from previous resync are removed from the queue at the same moment // as next resync inserts new requests. if !deleted { a.queue.AddRateLimited(key) </ins> } <del> a.queue.AddRateLimited(key) utilruntime.HandleError(err) </del> return true }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Resync period for HorizontalPodAutoscaler controller is set to 15 seconds in order to update HPAs every 15 seconds. Sometimes updates are skipped and period between two updates is 30 seconds. What you expected to happen: HPA should be updated every 15 seconds. How to reproduce it (as minimally and precisely as possible): Add debugs to HPA controller showing how often HPAs are processed. Create HPA. Anything else we need to know?: HPA controller uses rate RateLimitingQueue: New requests are always with 15 seconds delay: If request for the given HPA is in the queue then newly request is skipped. Requests are and removed from the queue at the same moment so there is a race condition. See following scenario: 00:00:00 - Resync adds request to queue to be executed at 00:00:15 00:00:15 - Resync adds next request to queue to be executed at 00:00:30 but it is skipped because previous request is still in the queue 00:00:15 - Request at 00:00:00 is executed 00:00:30 - Resync adds next request to queue to be executed at 00:00:45 00:00:45 - Request at 00:00:30 is executed In above scenario HPA wasn't updated between 00:00:15 and 00:00:45. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration: GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig autoscaling
/assign krzysztof-jastrzebski


Question: import ( "fmt" <ins> netutil "k8s.io/utils/net" </ins> "os" "runtime" "time"

In bare-metal IPv6 environment, any pod consuming storage resource is not able to run. The storage should be consumable, and the pod should become in status. Running the in a bare-metal IPv6 configurations. I have replaced the cloud-based storage resource with a mounted to a folder on my localhost. Steps alongside logs are below :- $kubectl get svc $kubectl describe svc nfs-server $kubectl describe pv nfs $kubectl get pod -w $kubectl get pod -w $kubectl describe pod nfs-busybox-6r96h I have built the master branch to make sure it was not fixed up to this moment.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig storage /tirage accepted /assign


Question: function set-windows-node-image() { WINDOWS_NODE_IMAGE_PROJECT="windows-cloud" if [[ "${WINDOWS_NODE_OS_DISTRIBUTION}" == "win2019" ]]; then <del> WINDOWS_NODE_IMAGE="windows-server-2019-dc-core-v20210112" </del> <ins> WINDOWS_NODE_IMAGE="windows-server-2019-dc-core-v20210413" </ins> elif [[ "${WINDOWS_NODE_OS_DISTRIBUTION}" == "win1909" ]]; then <del> WINDOWS_NODE_IMAGE="windows-server-1909-dc-core-v20210112" </del> <ins> WINDOWS_NODE_IMAGE="windows-server-1909-dc-core-v20210413" </ins> elif [[ "${WINDOWS_NODE_OS_DISTRIBUTION}" == "win2004" ]]; then <del> WINDOWS_NODE_IMAGE="windows-server-2004-dc-core-v20210112" </del> <ins> WINDOWS_NODE_IMAGE="windows-server-2004-dc-core-v20210413" </ins> elif [[ "${WINDOWS_NODE_OS_DISTRIBUTION,,}" == "win20h2" ]]; then <del> WINDOWS_NODE_IMAGE="windows-server-20h2-dc-core-v20210112" </del> <ins> WINDOWS_NODE_IMAGE="windows-server-20h2-dc-core-v20210413" </ins> else echo "Unknown WINDOWS_NODE_OS_DISTRIBUTION ${WINDOWS_NODE_OS_DISTRIBUTION}" >&2 exit 1

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Docker is always installed on node boot which adds significant start time and adds software that's unnecessary. containerd Windows nodes should not have Docker installed. Start a Kubernetes GCE Windows cluster via kube-up. You can run stuff like on it. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig windows /triage accepted


Question: return false } <ins> // This is O(N), but we expect serviceNodePorts to be small; // so small that we expect a linear search to be faster func containsNodePort(serviceNodePorts []ServiceNodePort, serviceNodePort ServiceNodePort) bool { for _, snp := range serviceNodePorts { if snp == serviceNodePort { return true } } return false } </ins> func CollectServiceNodePorts(service *api.Service) []int { servicePorts := []int{} for i := range service.Spec.Ports {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Re-applying a NodePort service that has two ports fails and crashes the API server: What you expected to happen: Do not crash, service is configured correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on VBox provider in macOS OS (e.g. from /etc/os-release): macOS/Minikube Kernel (e.g. ): Install tools: Others:
Seems that in func updateNodePorts, it ignored protocol when Detecting duplicate nodeport, I would have a fix on it :)
/sig network
FYI: I think this would be fixed in
/cc Maybe this is the time to have another review on and merge it :)
/assign
Fine, That makes sense

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /cc /assign What happened: a ClusterIP type service the service. Change it's type to NodePort and add a Port. Note: the new port has same port number but protocol with the existing Port. the updated service. The TCP Port was eaten. What you expected to happen: The updated service has both TCP and UDP ports. How to reproduce it (as minimally and precisely as possible): See the step above. Anything else we need to know?: If I create a NodePort service with the above two ports, it works. I write a test case to verify this case. I got a panic. The root cause is as below. We should take another look at this logic. BTW: changed some related code but it doesn't change the logic here. So I believe this bug is not introduced by . I also tested this on v1.5.1, it has the same issue. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I will be OOO the next week. Since this bug is not so urgent, I will fix it ASAP when I'm back. Folks who are interested in this can take it :) BTW, I can join in the discussion from my phone.
I'm not sure this is a bug. In the first configuration yaml, you're defining the ports array twice. I'm the second, you're only defining it once. The ports array should only be defined once.
Did you mean the below yaml by ? Please note that the two ports have same port number but different protocols. This is allowed.
Yes, that's the one I meant. But note you have "ports:" twice.
Ah, thanks for pointing it out! This might be the cause. But I remembered I also tested the case without the second line, it had the same issue. I can't verify this now. Will test it when I'm back to my keyboard.
I try the case without the second line, the issue also exists.

So I believe there is a bug here. Seems like there might be two issues here: a port is lost as is described in this issue. we shouldn't allow to configure two lines in the manifest. i.e. we should add validation for this case.
/remove-priority awaiting-more-evidence
can you please provide (or edit) example YAMLs then to allow for attempts at duplication?
Update: reproducing without the extra line. a ClusterIP type service the service (). Change it's type to NodePort and add a Port which has the same port number but different protocol with the existing Port. the updated service. We can see that the UDP Port was not created. I have dug into the code and will send a patch to fix it.
PR sent. Reviews welcome! Thanks!
There are/were bugs in the patch implementation wrt ports, because they need compound merge keys.
Yeah I had some discussios with in . But we also do have a bug in our service updating codes. See:


Question: // buildHandlerChain wraps the given handler with the standard filters. func buildHandlerChain(handler http.Handler, authn authenticator.Request, authz authorizer.Authorizer) http.Handler { requestInfoResolver := &apirequest.RequestInfoFactory{} <del> failedHandler := genericapifilters.Unauthorized(legacyscheme.Codecs) </del> <ins> failedHandler := genericapifilters.Unauthorized(scheme.Codecs) </ins> <del> handler = genericapifilters.WithAuthorization(handler, authz, legacyscheme.Codecs) </del> <ins> handler = genericapifilters.WithAuthorization(handler, authz, scheme.Codecs) </ins> handler = genericapifilters.WithAuthentication(handler, authn, failedHandler, nil) handler = genericapifilters.WithRequestInfo(handler, requestInfoResolver) handler = genericapifilters.WithCacheControl(handler)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kube-Scheduler exposes metrics in a secure manner (TLS and authentication required) on port by default. If one does not authenticate itself, the error message displayed is: What you expected to happen: Kube-Scheduler should emit a clear and understandable error message if authentication fails. Kube-Controller-Manager does that the following way: How to reproduce it (as minimally and precisely as possible): up a Kubernetes Cluster to access the exposed metrics endpoint (https://metrics-server:/metrics) without authentication Anything else we need to know?: xref: Environment: Kubernetes version (use ): 1.18.6 Cloud provider or hardware configuration: Bare metal OS (e.g: ): Debian GNU/Linux 10 (buster) Kernel (e.g. ): 4.19.0-9-amd64 Install tools: Kubeadm (Kube-Scheduler )
/sig scheduling instrumentation
If ok, I can try this fix /assign
/assign


Question: ETCD_QUORUM_READ: $(yaml-quote ${ETCD_QUORUM_READ}) EOF fi <ins> if [ -n "${CLUSTER_SIGNING_DURATION:-}" ]; then cat >>$file <<EOF CLUSTER_SIGNING_DURATION: $(yaml-quote ${CLUSTER_SIGNING_DURATION}) EOF fi </ins> else # Node-only env vars.

/kind e2e-test-failure GCE serial and GCI GCE serial started failing last evening https://k8s- https://k8s-
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/testing: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
https://k8s- is still green, I would suspect something ~last Friday afternoon gets merged
Here is what I found, after a while, the node lost their authorization to connect to API server. I believe this has the same root cause as
I am going to revert offending pr () to see if it helps. EDIT: Looks like is in the process of reverting it:

ref Test is failing on GCE slow and GCI GCE slow: https://k8s- https://k8s- Example failure https://k8s-
The test nodes appear to lose their authorization for accessing resources about 26m into the test. One of many "Unauthorized" errors starting at
, does this ring any bells ?
The certificates are all expiring:
I believe this is the regression introduced cc/ Can you guys take a look?
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. kind: Must specify exactly one of [, , ]. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
thanks. I'll open a PR to revert that.
Revert here


Question: return nil } <del> func (w *watchCache) getResourceVersion() uint64 { </del> <ins> func (w *watchCache) getListResourceVersion() uint64 { </ins> w.RLock() defer w.RUnlock() <del> return w.resourceVersion </del> <ins> return w.listResourceVersion </ins> } func (w *watchCache) currentCapacity() int {

sig-release-master-blocking ci-kubernetes-unit 05/30/2024 19:25 IST Prow Logs: Triage: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig api-machinery
/cc This started last week, but it seems to be fairly rare, so might be a consequence of something a bit earlier. The value of "2" suggests that it's happening on the first event, and my brain goes towards: - will you be able to take a look at it?
sure, I will have a look. /assign
We chatted with about that offline and it seems like a real correctness issue. Regression indeed introduced by We have a race with setting RV in watchcache and incoming events. The current explanation is: initializeds with RV=N event comes with RV=N+1 [but wasn't yet dispatched] polling introduced in that PR catches RV=N+1 then the event (with RV=N+1) is received
So it is a release blocker, then? (Upcoming alpha 1 release cut scheduled for tomorrow)
I don't know what the release policy is but we might have a potential fix for this issue.
This is a real correctness bug introduced 2 weeks ago. I'm not sure what's our policy is for alpha releases, but it sounds like a potential blocker to me.
The fix for it is already approved.


Question: PluginIsAttachable: false, // this field determines whether the plugin is attachable ReportedInUse:      true, } <del> oe.MountVolume(0 /* waitForAttachTimeOut */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */) </del> <ins> oe.MountVolume(0 /* waitForAttachTimeOut */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */, false /* isRemount */) </ins> } // Assert

TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet.

The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks!

Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command.

This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!

Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.

because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days: