Question: "Microsoft.Network/publicIPAddresses/pip1", }, { <del> desc: "ensurePublicIPExists shall update existed PIP's dns label", </del> <ins> desc:                    "ensurePublicIPExists shall update existed PIP's dns label", inputDNSLabel:           "newdns", foundDNSLabelAnnotation: true, </ins> existingPIPs: []network.PublicIPAddress{{ Name: to.StringPtr("pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: errors = append(errors, fmt.Errorf("--secure-port %v must be between 0 and 65535, inclusive. 0 for turning off secure port", s.BindPort)) } <ins> if (len(s.ServerCert.CertKey.CertFile) != 0 || len(s.ServerCert.CertKey.KeyFile) != 0) && s.ServerCert.GeneratedCert != nil { errors = append(errors, fmt.Errorf("cert/key file and in-memory certificate cannot both be set")) } </ins> return errors }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk.

I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security).

Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?


Question: package app import ( <ins> "errors" </ins> "fmt" "reflect" "runtime"

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? is:issue is:open ipv6 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): N/A, these are unit test changes in latest master Environment: Cloud provider or hardware configuration: Kubernetes container dev env on Ubuntu 16.04.1 OS (e.g. from /etc/os-release): Ubuntu 16.04.1 Kernel (e.g. ): 4.4.0-78-generic -Ubuntu x8664 GNU/Linux Install tools: Others: What happened: Upon inspection, no IPv6 test cases found in kubernetes/cmd/kube- What you expected to happen: Expect to see some IPv6 test cases exercised for kube-proxy server code (kubernetes/cmd/kube-). Also need some code coverage for negative conditions in How to reproduce it (as minimally and precisely as possible): Run unit tests in kubernetes/cmd/kube- Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig networking
/area ipv6
/sig network


Question: "//vendor/k8s.io/apimachinery/pkg/util/intstr:go_default_library", "//vendor/k8s.io/client-go/pkg/api:go_default_library", "//vendor/k8s.io/client-go/pkg/api/v1:go_default_library", <ins> "//vendor/k8s.io/client-go/pkg/apis/apps/v1beta1:go_default_library", </ins> "//vendor/k8s.io/client-go/pkg/apis/extensions:go_default_library", "//vendor/k8s.io/client-go/pkg/apis/networking:go_default_library", ],

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: return currentReplicas, utilization, rawUtilization, timestamp, nil } <ins> newReplicas := int32(math.Ceil(newUsageRatio * float64(len(metrics)))) if (newUsageRatio < 1.0 && newReplicas > currentReplicas) || (newUsageRatio > 1.0 && newReplicas < currentReplicas) { // return the current replicas if the change of metrics length would cause a change in scale direction return currentReplicas, utilization, rawUtilization, timestamp, nil } </ins> // return the result, where the number of replicas considered is // however many replicas factored into our calculation <del> return int32(math.Ceil(newUsageRatio * float64(len(metrics)))), utilization, rawUtilization, timestamp, nil </del> <ins> return newReplicas, utilization, rawUtilization, timestamp, nil </ins> } // GetRawResourceReplicas calculates the desired replica count based on a target resource utilization (as a raw milli-value)

What happened: The bug described in is still present in 1.14.8 When a change is made to the running deployment that requires a rolling update to be executed, both the current replicaset and the new replicaset is scaled out to max number of pods regardless of actual load. What you expected to happen: The bug to be fixed How to reproduce it (as minimally and precisely as possible): Follow this guide The cluster does NOT have to be an EKS cluster, the bug is not related to the underlying platform or infrastructure. Anything else we need to know?: This is bug is really critical, and should be addressed and hotfixed as soon as possible. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-15T12:02:12Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Container Linux by CoreOS 2247.5.0 (Rhyolite) Kernel (e.g. ): 4.19.78-coreos Install tools: kube-aws v.0.14.2 ()
This bug does only apply to the and API version, not the API.
same issue i think it was really critical iuuse so hope hot fixed asap
/sig autoscaling
I've been looking into this and have made some interesting observations about the behaviour within our systems at least: Firstly, during the update, we see even though is greater in size than the current replica count, and this continues to increase over time. This suggests the bug may be somewhere in here: is set to and so I believe is being increased (to greater than ) by the method. Secondly, we see the scale doubling (approximately) during the rolling update.

The observation I've made it that it seems to be getting updated to the total count of the old and new replicasets pods combined, as if it is using the number of pods that match the label matcher for the deployment as the desired count when it finally updates the scale. I have yet to find a piece of code that verifies this though, merely an observation. Hopefully this helps narrow down the bug, it may also be a wild goose chase I can do more testing if anyone has any ideas, just let me know
Just for information, I am seeing similar behaviour using a custom metric, not CPU - I am using . In our dev environment the config is of 1, of 3. Even when there are no requests coming in, hpa is scaling to 3 or 4 pods then cooling back down to 3 pods after 5 minutes.
CPU Utilization of Pod-1 is . CPU Utilization of Pod-2 is . So CPU Utilization of Deployment is . So calculated number of replicas is . But is , so as a result, new number of replicas is . New Pod-3 is created due to RollingUpdate with and . CPU Utilization of Pod-1 is . CPU Utilization of Pod-2 is . CPU Utilization of Pod-3 is because metrics-server can't provide metrics of freshly maded Pod-3 and hpa-controller treats pod's resource utilization as when deployment tend to scale in. So CPU Utilization of Deployment is . So new number of replicas is in spite of .
I'd like to fix this issue in .
as workaround, can we just set to solve it?
, just to make sure this is not forgotten Specifying also results in an uncontrolled roll out. Please verify that the fix in apply to the and scenario as well. Or is your about just a typo? Your PR actually specifies , which is what I reported in this bug.
, if I'm not mistaken this bug is related to , not . So specifying won't help. Edit: There is actually no such thing as a , which wouldn't make any logical sense either.
Sorry, I meant but copy pasted the incorrect text. I think is simply a typo. So, as workaround, can we just set to solve it?
Unfortunately is really not a good workaround.

Say you have a deployment with 200 pods, would slow down the rolling update dramatically, as only one new pod could be created at a time, instead of all 200 at once. If the startup time for your pod is 10 seconds, replacing all 200 pods would take 10 seconds with , but 2000 seconds (33.3 minutes) with And according to the documentation, setting to anything except for 0 will reduce the number of pods (basically reducing capacity) during a rolling update. We cannot tolerate any reduced capacity during a rolling update, as the service powered by the pods might get saturated with incoming customer requests. Example: when this is set to 30%, the old ReplicaSet can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old ReplicaSet can be scaled down further, followed by scaling up the new ReplicaSet, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods. My understanding is that the setting is to ensure that your cluster (worker nodes) are not saturated during a rolling update. In our case we do have enough free capacity in the cluster to always be able to immediately replace all pods for any running service.
I'm sorry. is typo. is correct.
Unfortunately no. When I try scenario described in with , HPA scale deployment regardless of current load. become depends on timing.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
What is the status of this issue? I still believe it's a critical bug that needs to be addressed.
+1
We may be running into the same or a problem that may be connected to this. We run a deployment with The HPA, which targets the deployment, is set to scale on a custom metric which is scraped every 30 seconds (as the value is a calculated as a it takes 60 seconds for the first value to appear).

When the rolling update runs, even with completely idle deployment at 1 replicas, the HPA starts to add replicas to the deployment until it either hits the max replicas (which is 8 for this deployment) or until the first Pod generates a metric. The in method works as intended. It uses 100% of the target value for pods missing metrics, so, with 2 replicas running, one at 100% and the other one at 1% it still returns 2 replicas as the desired replicas count. I believe the problem is related to how the current number of replicas is taken from the deployment spec during HPA reconciliation (). During a rolling update, we saw the deployment spec and the deployment status being out of sync for long periods of time (over 3 minutes). This does not always happen, but it happens most of the times we do a rolling update. The HPA status is also confusing, as it indicates that the metric is above target when in fact is well below target: Should the HPA reconciliation use the current number of replicas from the status rather that from the spec?
thanks for the report. I am able to reproduce this on 1.14.10 with your test yaml. I will find the root cause and come back to you shortly.
/assign josephburnett
This issue is reproduced in my environment (v1.15.9) too, and it's resolved by replacing the metric type from to . I suspect the formula calculating the average is wrong.
I've cherry picked on top of 1.14.8 and it indeed fixes the bug. I've approved the PR and requested to fix it in as well.
, thx a lot! So in the next release of 1.14, 1.15, 1.16 and 1.17 this bug will be fixed?
Unless I'm mistaken this won't be cherry-picked back to 1.14 as it's fallen out of support as of the GA release of 1.17 as per . At best it'll be getting back to 1.15.
I'll work on backporting this to 1.15 - 1.18.
Cherry pick process (for reference): 1:15: 16: 17: 18:
This bug is affecting us badly , Is this fix merged in 1.16.x release ?

if not is there a temporary solution to fix this issue
This fix is available from 1.16.9.


Question: // testPod creates a pod that consumes a pv and prints it out. The output is then verified. func (t *PersistentVolumeUpgradeTest) testPod(f *framework.Framework, cmd string) { <del> pod := framework.CreatePod(f.ClientSet, f.Namespace.Name, []*v1.PersistentVolumeClaim{t.pvc}, false, cmd) </del> <ins> pod := framework.MakePod(f.Namespace.Name, []*v1.PersistentVolumeClaim{t.pvc}, false, cmd) </ins> expectedOutput := []string{pvTestData} f.TestContainerOutput("pod consumes pv", pod, 0, expectedOutput) }

https://k8s- This is upgrading from the latest 1.6 branch to master.
If you're not the right person to look into this, please find someone who can.
PR changed the MakePod() call to CreatePod() instead, which I don't think is correct because TestContainerOutput() will already create the pod. what was the reason for changing it?
Hmmm although it looks like the upgrade test was failing even before then, so maybe that's just a red herring.
Actually the test started failing consistently with the same pv_util signature after 3/31, which coincides with the change to call CreatePod(). So I think yes, that is the reason for the latest failures. I will work on a change to move it back to MakePod(). However, the upgrade test in general has been failing for a while. But the signatures have been different. Here's the signature before 3/31: https://k8s-
we updated MakePod to allow passing multiple claims to pod spec. As part of review comment I addressed this change. in the previous code, pod spec has MountPath for volume set to : "/mnt", in the new code we set the mount path to "/mnt/volume1"


Question: return fmt.Errorf("pod %s is not running and cannot be attached to; current phase is %s", p.PodName, pod.Status.Phase) } <del> containerName := p.ContainerName if len(containerName) == 0 { glog.V(4).Infof("defaulting container name to %s", pod.Spec.Containers[0].Name) containerName = pod.Spec.Containers[0].Name } </del> // TODO: refactor with terminal helpers from the edit utility once that is merged var stdin io.Reader tty := p.TTY

runs foo and then waits for it to be ready or terminated. If the pod is terminated, it prints the log, and if it's not terminated, it attaches the output. However, it does not handle the case that the pod terminates between returning the ready status and attaching, as can be the case in the : Options to fix this: Fix kubectl: If attach fails, fall back to printing the logs If attach fails, check the status again and if it's terminated, print the logs Fix the flaky test: Prevent the container from exiting (e.g. `sh -c 'echo running in container && sleep 30') Prevent the container from entering ready state (e.g. )
Thanks for debugging. I think we should probably do both: If attach fails and the container is now terminated, fall back to printing logs Make the e2e more robust by not exiting for a while to ensure time to attach. I'm happy to review PRs


Question: "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/wait" fileutil "k8s.io/kubernetes/pkg/util/file" <ins> "k8s.io/kubernetes/pkg/util/mount" </ins> "k8s.io/kubernetes/pkg/util/node" "k8s.io/kubernetes/pkg/volume" volutil "k8s.io/kubernetes/pkg/volume/util"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The method for ceph rbd volume plugin generates the path with code as follows: So the device mount path's format will be: , this implementation has two problems: with other volume plugins, as all the other plugins return , see as an example. method of ceph rbd plugin is not working properly, because rbd plugin uses method to find the device mount path(just like all the other plugins do), and method assumes the device mount path's format is (see ). What you expected to happen: The device mount path of rbd plugin should be . Anything else we need to know?: Do we have any reason for this design? If not, I will submit a PR to fix this, the adjusted method maybe look like this: /sig storage
/assign
nice catch There are two ways to fix this: Change mount path. But this will introduce compatibility issues during upgrade. Use , instead of to replace PR welcome
cc
Hi , there is a difference between and : returns the device path, such as , and returns the device mounted path, such as . What we really want is the second one, as we need to find the pool and image name. So we can't just simply replace with . Maybe we can find the mounted path specially for rbd plugin, use , the code looks like this:
thanks let make a forward moving plan: fix and append to the global path keep in . If no path is found, fall back to old path scheme using your code
Nice idea , I've submit a PR for this, PTAL.
I've checked and the reason is that I tested with the old release branch, the has been changed with in mount path, and causes the problem. thanks for your PR
i read again. There is another return path . So in rbd, the missing dir will not make into , but the second return should still give us the dir we need
You are right , indeed has two return path, but the second return still can't meet our needs. Because the parameter of is the mount path for a specific pod, such as , its format is , so will return the volume name. This volume name is returned by (see ), usually is the PV's name or the name specified in , both of these are not .


Question: totalPodCapacity += podCapacity.Value() } <del> currentlyScheduledPods := framework.WaitForStableCluster(cs, masterNodes) </del> <ins> currentlyScheduledPods := WaitForStableCluster(cs, masterNodes) </ins> podsNeededForSaturation := int(totalPodCapacity) - currentlyScheduledPods ginkgo.By(fmt.Sprintf("Starting additional %v Pods to fully saturate the cluster max pods and trying to start another one", podsNeededForSaturation))

What would you like to be : is used at the beginning of some e2e scheduler tests to wait until the cluster is in a stable state. It currently waits until all existing pods are scheduled, but it should also wait until pods from previous tests are actually deleted. We can do this by assuming that pods should exist only in the kube-system namespace, and so WaitForStableCluster should wait until there are no pods in any other namespace. Why is this needed: /help /sig testing /sig scheduling
/milestone v1.17 /priority critical-urgent
/cc
I'll take this one, can't figure out how to get ci bot to assign to me...
/assign
/remove-help


Question: -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS COMMIT ` <ins> assert.Equal(t, []*netutils.LocalPort{ { Description: "nodePort for ns1/svc1:p80", IP:          "", IPFamily:    netutils.IPv4, Port:        svcNodePort, Protocol:    netutils.TCP, }, }, fp.portMapper.(*fakePortOpener).openPorts) </ins> assertIPTablesRulesEqual(t, expected, fp.iptablesData.String()) }

I've been following the official docs for kubeadm and how to set up a new cluster. The cluster is up and running but I noticed warning events when creating node port services: However, the node port does work (I see the output when accessing node ip:nodeport). Might it be related to something else than Kubernetes? There is no such warning event. Set up a fresh cluster with kubeadm and then create a service of nodeport type. No response <details</details<detailsHetzner cloud </details<details</details<detailscloud-init </details<detailscontainerd </details<detailscni=cilium csi=hetzner </details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
check with netstat or ss if there is a process listening on that port
no, I specified exact random port e.g. and checked it before with ss - it was not being used. Then I created the service, the warning occured, and after that the port was taken:
kube-proxy binds the NodePort to warn users that they may be shadowing a process, but it logs the errors if it can not bind the port
I met the same error and I confirmed the node port was not used by any other process. After looking into the code, I think this is a bug in kube-proxy that it would try to open a node port twice with same (empty) IP and port when is empty. contains and in below code.: I have created to fix it.
Awesome find. Did you figure out a work around?

I am encountering this issue with this version of the image:
The error looked scaring but the nodePort should be working, as it was just caused by a duplicate port opening attempt. v1.23.5 and later releases have the fix.


Question: pod.Spec.HostNetwork = true _, err := cs.CoreV1().Pods(ns).Create(pod) framework.ExpectNoError(err) <del> framework.ExpectNoError(f.WaitForPodRunning(podName)) </del> <ins> framework.ExpectNoError(f.WaitForPodReady(podName)) </ins> framework.Logf("Netexec pod %q in namespace %q running", podName, ns) defer func() {

Per PR the test fails to error properly. One line of particular interest from that PR: If you wonder why it does not fail on docker, that's because the test does not report the error correctly but fails the same way. This issue is intended to track the discussion and resolution of making this test error properly so that it is useful to CI Signal /sig networking /cc
The label(s) cannot be appled. These labels are supported: <detailsIn response to : Per PR the test fails to error properly. One line of particular interest from that PR: If you wonder why it does not fail on docker, that's because the test does not report the error correctly but fails the same way. ` /cc </details
I'm following up on this in
/sig network


Question: # Heapster <del> Heapster enables monitoring of Kubernetes Clusters using [cAdvisor](https://github.com/google/cadvisor). Detailed information about heapster can be found [here](https://github.com/GoogleCloudPlatform/heapster). </del> <ins> Heapster enables monitoring of Kubernetes Clusters using [cAdvisor](https://github.com/google/cadvisor). The kubelet will communicate with an instance of cAdvisor running on localhost and proxy container stats to Heapster. Kubelet will attempt to connect to cAdvisor on port 4194 by default but this port can be configured with kubelet's `-cadvisor_port` run flag. Detailed information about heapster can be found [here](https://github.com/GoogleCloudPlatform/heapster). </ins>

Currently, kubelet uses a for communicating with cAdvisor. , were having an issue because we didn't realize making port available was required for kubernetes.
I prefer not do this since we plan to make 4194 publically reserved port for cAdvisor. Also cAdvisor is the default deamon running on every node of a kubernetes cluster. The issue you encountered sounds like a bad document issue we had for cAdvisor. We should fix the document instead of allow the port being configurable.
can you expound on 'bad document' issue?


Question: # Stop docker before making these updates systemctl stop docker <ins> # Install openvswitch yum install -y openvswitch systemctl enable openvswitch systemctl start openvswitch </ins> # create new docker bridge ip link set dev ${DOCKER_BRIDGE} down || true brctl delbr ${DOCKER_BRIDGE} || true

It appears that the provision-network script that is run on the kubernetes-master can cause salt provisioning on the minions to fail or block in a forever retry timeout error loop. I think it's because of the service network.restart that happens after openvswitch is installed in provision-network. I only say this because when I disable installing docker on kubernetes-master in vagrant, I do not have an issue. - is there a different way we can structure this on the kubernetes-master? For example, could we run the script in or something?


Question: "networking.go", "node_lease.go", "pods.go", <ins> "podtemplates.go", </ins> "privileged.go", "projected_combined.go", "projected_configmap.go",

This issue proposes the writing of a new test for the API endponts , , , and with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these results from APIsnoop, we are lacking in test coverage and conformant tests for PodTemplate. We chose these three following endpoints for this mock test: - - Outline: PodTemplates, Expect no PodTemplates a PodTemplate (1), Expect no error PodTemplate (1), Ensure the same as how it was created PodTemplate (1) with label a=1, Expect no error PodTemplate (1), Expect label in returned data to be present PodTemplates by label a=1, Expect one PodTemplate PodTemplate (1) Expect no error PodTemplates, Expect no PodTemplates Fetch dependencies: This mock test is using client-go's standard functions to talk to the PodTemplate part of the API. If a test with these calls gets merged in, conformance will go up by 3 endpoints. /sig architecture /area conformance
We decided not to deprecate. We can link to that, but otherwise this is OK to write / promote.
ref: for the decision not to deprecate
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: k8sruntime "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes/scheme" <ins> restclient "k8s.io/client-go/rest" </ins> ref "k8s.io/client-go/tools/reference" )

What happened: kube-scheduler panics when client-go tries to finish the existing event series, see: What you expected to happen: kube-scheduler should not panic How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority critical-urgent /sig scheduling


Question: return nil } <ins> func (cephfsMounter *cephfsMounter) checkFuseMount() bool { execute := cephfsMounter.plugin.host.GetExec(cephfsMounter.plugin.GetPluginName()) switch runtime.GOOS { case "linux": retBytes, err := execute.Run("/bin/ls", "/sbin/mount.fuse.ceph") if err == nil && string(retBytes) == "/sbin/mount.fuse.cephn" { glog.V(4).Infof("/sbin/mount.fuse.ceph exists, it should be fuse mount") return true } return false } return false } func (cephfsVolume *cephfs) execFuseMount(mountpoint string) error { // cephfs keyring file keyring_file := "" // override secretfile if secret is provided if cephfsVolume.secret != "" { // TODO: cephfs fuse currently doesn't support secret option, // remove keyring file create once secret option is supported. glog.V(4).Infof("cephfs mount begin using fuse.") keyringPath := cephfsVolume.GetKeyringPath() os.MkdirAll(keyringPath, 0750) payload := make(map[string]util.FileProjection, 1) var fileProjection util.FileProjection keyring := fmt.Sprintf("[client.%s]n", cephfsVolume.id) + "key = " + cephfsVolume.secret + "n" fileProjection.Data = []byte(keyring) fileProjection.Mode = int32(0644) fileName := cephfsVolume.id + ".keyring" payload[fileName] = fileProjection writerContext := fmt.Sprintf("cephfuse:%v.keyring", cephfsVolume.id) writer, err := util.NewAtomicWriter(keyringPath, writerContext) if err != nil { glog.Errorf("failed to create atomic writer: %v", err) return err } err = writer.Write(payload) if err != nil { glog.Errorf("failed to write payload to dir: %v", err) return err } keyring_file = path.Join(keyringPath, fileName) } else { keyring_file = cephfsVolume.secret_file } // build src like mon1:6789,mon2:6789,mon3:6789:/ hosts := cephfsVolume.mon l := len(hosts) // pass all monitors and let ceph randomize and fail over i := 0 src := "" for i = 0; i < l-1; i++ { src += hosts[i] + "," } src += hosts[i] mountArgs := []string{} mountArgs = append(mountArgs, "-k") mountArgs = append(mountArgs, keyring_file) mountArgs = append(mountArgs, "-m") mountArgs = append(mountArgs, src) mountArgs = append(mountArgs, mountpoint) mountArgs = append(mountArgs, "-r") mountArgs = append(mountArgs, cephfsVolume.path) glog.V(4).Infof("Mounting cmd ceph-fuse with arguments (%s)", mountArgs) command := exec.Command("ceph-fuse", mountArgs...) output, err := command.CombinedOutput() if err != nil || !(strings.Contains(string(output), "starting fuse")) { return fmt.Errorf("Ceph-fuse failed: %vnarguments: %snOutput: %sn", err, mountArgs, string(output)) } return nil } </ins> func getVolumeSource(spec *volume.Spec) ([]string, string, string, string, bool, error) { if spec.Volume != nil && spec.Volume.CephFS != nil { mon := spec.Volume.CephFS.Monitors

The goal of this issue is to design a CephFS driver using FUSE client for Kubernetes.</brExisting Ceph kernel client has the following drawbacks:</br• Kernel client does not support client-enforced quotas. • Kernel CephFS doesn't support quota or capacity, capacity requested by PVC is not enforced or validated. • Kernel CephFS doesn't work with SELinux, setting SELinux label in Pod's security context will not work.
Just a detail. Since we will rely on ceph-fuse, we will need the ceph-fuse client to be available on all kubernetes nodes. We can add a parameter to allow the use of the kernel client as well. I would say better make the fuse client opt-in and keep the kernel one as the default option. The kernel client might support quotas in the future. Thanks
/sig storage
We are also inclined to use fuse instead of kernel client ,any progress on this issue?
I like to continue this job and prepare to send a PR to make kubernetes support cephfs fuse mounting.
/assign I think this task would be valuable. =)
waiting for the PR, do you have a plane to support s3 fuse ?
No, no plan for s3fs-fuse currently
also see and cc
Ceph-fuse support in the command to display the specified secret, it may take some time to support, because it involves LibFUSE(I will follow up this demand). On the other hand, ceph-fuse has stability, quota function, expansion, upgrade and other advantages. Although ceph-fuse only supports the secret file way, but if it can be pushed to the k8s community, there is a very meaningful thing.


Question: options = metav1.NewDeleteOptions(int64(gracePeriod)) } options.OrphanDependents = &orphan <del> return deleteResource(info, out, shortOutput, options) </del> <ins> return deleteResource(info, out, shortOutput, options, gracePeriod) </ins> }) if err != nil { return err

IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy


Question: }, { APIGroups: []string{""}, <del> Resources: []string{"persistentvolumesclaims"}, </del> <ins> Resources: []string{"persistentvolumeclaims"}, </ins> Verbs:     []string{"get", "list", "watch", "update"}, }, {

Current CSI E2E test has some issues which prevent it to run successfully the a local cluster. /sig storage /kind bug /assign


Question: c.supportsSELinux, err = c.kubeVolHost.GetHostUtil().GetSELinuxSupport(dir) if err != nil { <del> klog.V(2).Info(log("error checking for SELinux support: %s", err)) </del> <ins> // The volume is mounted. Return UncertainProgressError, so kubelet will unmount it when user deletes the pod. return volumetypes.NewUncertainProgressError(fmt.Sprintf("error checking for SELinux support: %s", err)) </ins> } if !driverSupportsCSIVolumeMountGroup && c.supportsFSGroup(fsType, mounterArgs.FsGroup, c.fsGroupPolicy) {

On a busy system, kubelet failed to tell the container runtime that a pod's CSI volume supports SELinux and should be relabeled with pod's SELinux label. The pod then can't access the volume. Kubelet correctly detects that the CSI volume supports SELinux and tells the container runtime to relabel it. Unknown. Get a busy system and try hard? CSI volume plugin tried to detect that the volume supports SELinux during SetupAt. This check failed: Current CSI volume plugin code silently consumes the error and assumes that the volume does not support SELinux at all: Therefore, the volume is marked as mounted and given to the container runtime without the right SELinux options and the runtime does not relabel volume. <details22.0 </details<detailsNone </details<details</details<details</details<detailsCRI-O, but I think it's not really relevant. </details<details</details
/sig storage /triage accepted
We've updated kubelet in our clusters to v1.20.13 (which contains the cherry picked fix), but still observe the issue


Question: # Entry provided via env var to simplify invocation. if [[ -n ${E2E_USE_GO_RUNNER:-} ]]; then set -x <del> /gorunner exit $? </del> <ins> /gorunner && ret=0 || ret=$? exit ${ret} </ins> fi # We get the TERM from kubernetes and handle it gracefully

Same as an issue that occurred for v1.14.0: the failing test causes ginkgo to exit non-zero which causes the script to abort without running the routine. A recent PR to this script fixed a race but we didn't realize that now that the command was running in the foreground (and not the background) that the non-zero status would impact that script. Currently, Sonobuoy can get around this by just using the Golang based runner via . The Go-runner does not seem to have a similar vulnerability. This is going to be the default for Sonobuoy runs starting in Sonobuoy v0.16.0 if the PR gets merged. Honestly, I think that we may want to consider just removing the bash script all-together for v1.17.0 if the go runner works well for this entire cycle. See the original reporting of the issue and diagnosis here: /sig testing /area conformance /priority important-soon
/assign The PR in question that re-emerged this problem: FWIW Sonobuoy is adding an integration suite that will be able to test this going forward (soon). I think we need to fix the script (and get it cherrypicked into v1.16.x branches) I think that if the go runner proves stable over v1.16 then we default to it going forward We consider removing entirely if we think we can do that without impacting other workflows. Maybe that wont be possible due to other workflows though. cc
is probably never called due to , it can probably be fixed by adding a or before line 70.
Yes that is exactly why it is but in the past when I recommended removing/modifying the errexit code I've gotten pushback from a googler just due to convention: see Just will add something like to the lines that need this


Question: "STANDARD_E8S_V3":        16, "STANDARD_E8_V3":         16, "STANDARD_E96AS_V4":      32, <ins> "STANDARD_E96A_V4":       32, </ins> "STANDARD_F1":            4, "STANDARD_F16":           64, "STANDARD_F16S":          64,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: nodeName := pod.Spec.NodeName gomega.Expect(nodeName).NotTo(gomega.BeEmpty(), "pod.Spec.NodeName must not be empty") <del> ginkgo.By(fmt.Sprintf("[init] waiting until the node=%s is not using the volume=%s", nodeName, pv.Name)) </del> <ins> // Snapshot tests are only executed for CSI drivers. When CSI drivers // are attached to the node they use VolumeHandle instead of the pv.Name. volumeName := pv.Spec.PersistentVolumeSource.CSI.VolumeHandle ginkgo.By(fmt.Sprintf("[init] waiting until the node=%s is not using the volume=%s", nodeName, volumeName)) </ins> success := storageutils.WaitUntil(framework.Poll, f.Timeouts.PVDelete, func() bool { node, err := cs.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) framework.ExpectNoError(err) volumesInUse := node.Status.VolumesInUse framework.Logf("current volumes in use: %+v", volumesInUse) for i := 0; i < len(volumesInUse); i++ { <del> if strings.HasSuffix(string(volumesInUse[i]), pv.Name) { </del> <ins> if strings.HasSuffix(string(volumesInUse[i]), volumeName) { </ins> return false } } return true }) framework.ExpectEqual(success, true) } cleanup := func() {

The volume snapshot controller tests can result in an empty VolumeSnapshot being taken if NodeUnstageVolume occurs after the VolumeSnapshot is taken. Previously there was a change made in Windows to ensure that NodeUnstageVolume finishes before continuing with the test; however, this needs to also to be included in other distros as well. NodeUnpublish, which removes the bind mount, does not force the caches to flush. Tests - The CI jobs should pass when running with a non-hostpath driver. I'm uncertain on why we've seen an increase in these failures, but running the following tests with a non-hostpath driver appear to flake regularly: [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion In these tests we've seen the following with the AWS EBS CSI Driver. Driver logs on the node: In the driver logs (in the controller pod) I can see:
/sig storage
/triage accepted /priority important-soon all snapshot tests with real CSI drivers are quite flaky, I saw it at least in AWS, Azure and GCE in a downstream CI job


Question: } // TODO use generalized labels once they are implemented (#341) <del> b := resource.NewBuilder(mapper, typer, factory.ClientMapperForCommand(cmd)). </del> <ins> b := resource.NewBuilder(mapper, typer, factory.ClientMapperForCommand()). </ins> NamespaceParam(cmdNamespace).DefaultNamespace(). SelectorParam("kubernetes.io/cluster-service=true"). ResourceTypeOrNameArgs(false, []string{"services"}...).

I think the need for this argument to the factory methods has passed - it means that callers who want to provide configurable behavior have to do it through the factory itself, which is acceptable. We can remove it whenever.
I removed most of the unused arguments. But, There are a few methods still using the argument. Namely, , , and . Before proceeding any further I wanted to ask what should be done about these methods? Deprecated/Refactored/Removed ?
The factory methods were the most important. The pattern before was people using the flags by name (cmd.Flags().Get("foo")), but now factory methods must use pointer flag vars, so there's nothing that should need the command there. The printer methods are optional units that can be used when you add the flags for them to your command. Unless we refactor the definition of those flags to use pointers and return a printer struct (something I had started a while back and never finished), it's not worth changing them right now. The other consumer of factory is OpenShift - we wrap the factory to add our additional types (implementations of things like describers, action verbs, etc): - but we will pick up and remove use of cmd when we rebase.
Thanks for the explanation. I've cleaned the as well. Print method APIs are untouched as you suggested.
cc
Looks like this was done by PR


Question: } } } <ins> for i := range podSpec.Containers { DropDisabledVolumeMountsAlphaFields(podSpec.Containers[i].VolumeMounts) } for i := range podSpec.InitContainers { DropDisabledVolumeMountsAlphaFields(podSpec.InitContainers[i].VolumeMounts) } } // DropDisabledVolumeMountsAlphaFields removes disabled fields from []VolumeMount. // This should be called from PrepareForCreate/PrepareForUpdate for all resources containing a VolumeMount func DropDisabledVolumeMountsAlphaFields(volumeMounts []api.VolumeMount) { if !utilfeature.DefaultFeatureGate.Enabled(features.MountPropagation) { for i := range volumeMounts { volumeMounts[i].MountPropagation = nil } } </ins> }

says "Before persisting the object to storage, clear disabled alpha fields". There is that actually implements it but there are some discussions about it. This issues is a reminder that introduced by needs to be cleared too when the discussion in settles down. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
kind/cleanup priority/important-soon
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details


Question: } } <del> func (t *timeBudget) takeAvailable() time.Duration { </del> <ins> func (t *timeBudgetImpl) takeAvailable() time.Duration { </ins> t.Lock() defer t.Unlock() result := t.budget

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: ctx, closeTransformers := context.WithCancel(serverCtx) defer func() { <del> // TODO: increment success metric when updatedEffectiveConfig=true </del> // TODO can work queue metrics help here? if !updatedEffectiveConfig { // avoid leaking if we're not using the newly constructed transformers (due to an error or them not being changed) closeTransformers() } <ins> if updatedEffectiveConfig && err == nil { metrics.RecordEncryptionConfigAutomaticReloadSuccess() } </ins> if err != nil { <del> // TODO: increment failure metric </del> <ins> metrics.RecordEncryptionConfigAutomaticReloadFailure() </ins> utilruntime.HandleError(fmt.Errorf("error processing encryption config file %s: %v", d.filePath, err)) // add dummy item back to the queue to trigger file content processing. d.queue.AddRateLimited(key)

ref - /sig auth
/triage accepted
/retitle encryption-at-rest: add success or failure metric when encryption configuration attempts to dynamically update
/assign


Question: newCfg := make(map[string]string) newCfg[cfgAccessToken] = token.token.AccessToken newCfg[cfgRefreshToken] = token.token.RefreshToken <ins> newCfg[cfgEnvironment] = token.environment </ins> newCfg[cfgClientID] = token.clientID newCfg[cfgTenantID] = token.tenantID newCfg[cfgApiserverID] = token.apiserverID

What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } func tryDecodePodList(data []byte, defaultFn defaultFunc) (parsed bool, pods api.PodList, err error) { <del> obj, err := api.Scheme.Decode(data) </del> <ins> json, err := utilyaml.ToJSON(data) if err != nil { return false, api.PodList{}, err } obj, err := api.Scheme.Decode(json) </ins> if err != nil { return false, pods, err }

Following on from the Kubelet should accept YAML pod specifications. Currnelty it supports only JSON pod specifications.


Question: errNotMounted = "not mounted" ) <ins> var ( // Error statx support since Linux 4.11, https://man7.org/linux/man-pages/man2/statx.2.html errStatxNotSupport = errors.New("the statx syscall is not supported. At least Linux kernel 4.11 is needed") ) </ins> // Mounter provides the default implementation of mount.Interface // for the linux platform.  This implementation assumes that the // kubelet is running in the host's root mount namespace.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We saw this several times: after restarting kubelet, kubelet didn't creating new Pods. The healthz endpoint showed that the was stuck: From the stack trace, the goroutine was stuck in the call of when doing , so no other updates can be processed: Listing the volumes of orphaned Pods on this Node, there was an unresponsive NFS volume, stat it will hang forever, so the above goroutine should be stuck in cleaning up this Pod: The NFS server IP is a service IP, whose backend Pod happened to be scheduled to this Node. However, as was stuck, it didn't handle Pod Add/Update from apiserver and didn't create the backend Pod, hence a dead lock. Searched PR and issue history, it seems there were several ones fixing NFS volume issues but looks like not covering this case: (however reverted) (still open) Is there anyone still working on it? The main syncloop of kubelet shouldn't hang due to an unresponsive NFS volume, leading to no Pods can be created. Kubernetes version (use ): v1.20.5
/sig storage /sig node similar with and And the problem code is in According to and 's comments in , it will be removed later, but I checked the new code in mount-utils. The func is not changed: Possible fix to this would be something like below:
/assign
Hey Quick question: the links that you attached that could be possible fixes, were really helpful! Since the approach to check for wouldn't help here, a timeout based approach seems best, I'm a little unsure about what this timeout period could be for this case, any suggestions on the same?
I have no idea. you may ask sig-node guys in the slack or weekly meetings. Or send a PR and discuss it in the PR. The only nfs related timeout that I can found is 1 minute in
hit again, it‘s easy to reproduce if there is nfs service in the cluster.
/assign /triage accepted
/area node-lifecycle
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue New contributor can follow some advice in .
Hey ! I'm wondering if this issue is still being worked on? I'd like to help if there's room to do so!
I'm not working on it. Thanks for the help.
awesome!
/assign
Hi are you still working on this?

I'm interested in this and please let me know if I can give it a try, Thanks! : )
I'm actually still on it but if you're open to it we can team up? I don't mind sharing
Thanks for reply, I am vary glad to hear that, I think you can continue your work and I would take a look at other issues, if there is anything I can help, please feel free to ping me : )
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Is someone still working on this or I am free to take a stab at this? I have ran into this a few times and trying to fix this cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
/remove-lifecycle rotten /remove-label good-first-issue This isn't a great first issue to work on. It requires having a NFS server and instrumenting it to be down to test the fix.
The label(s) /remove-label good-first-issue api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feedlabels -additionallabelslabels -? <detailsIn response to : /remove-lifecycle rotten /remove-label good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-good-first-issue
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the PR is closed You can: Mark this PR as fresh with Close this PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Does this problem still exist?
/triage needs-information
/remove-lifecycle stale
/triage accepted cancel
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted cancel Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
remove-triage accepted
/remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
statx with ATSTATXDONT_SYNC flag will not stuck in syncLoop. how about replace stat with statx?


Question: UpdateFunc: func(prev, obj interface{}) { prevNode := prev.(*v1.Node) newNode := obj.(*v1.Node) <del> if newNode.Labels[v1.LabelTopologyZone] == prevNode.Labels[v1.LabelTopologyZone] { return } </del> az.updateNodeCaches(prevNode, newNode) }, DeleteFunc: func(obj interface{}) {

Doing a micro upgrade from one level of 1.23 to another level of 1.23, we noticed disruption to PDB based app went from about 5s to about 13s when compared to a 1.22.z to 1.22.z+1 update. I did not expect any disruption. This one is a bit difficult, we have thousands of runs to compare averages and distributions against. It's specifically azure, it's specifically the service load balancer, it's worst on 1.23 to 1.23 upgrades versus 1.22 to 1.23 upgrades. We tracked it down to one of these three commits: When we revert all three, we see improved behavior (our test was in ). We attempted to workaround by setting the exclude-from-lb label, but this filter prevents the change from being observed by the informer's cache: When we attempted to remove the filter entirely (attempt here ), we get from <details</detailsAzure <details</details<details</details<details</details<details</details
quoting from on an internal thread klog.V(2).Infof("reconcileLoadBalancer for service (%s)(%t): lb backendpool - found unwanted node %s, decouple it from the LB", serviceName, wantLb, nodeName)
/sig cloud-provider
ping
/cc
Adding more details to the issue... We analyzed how Azure Load Balancer backends are modified as cluster nodes become or stop being available, which in our tests can happen in two ways: by labeling nodes withby rebooting nodes With kubernetes prior to this PR (, solving ), namely kubernetes 1.22, shipped with openshift 4.9, we notice that: cloud service controller in receives event notifications on nodes and updates accordingly the list of available nodes for load balancers (): a node is excluded from a load balancer backend pool if it has the "exclude" label above, or if its status is not ; in receives the updated list of nodes from the cloud service controller this list differs from the current list of nodes in the backend pool for a given load balancer, it removes spurious nodes and adds any new nodes. particular, node deletion begins here: and takes place in the function.

So far so good, if I label a node that I want to exclude from a load balancer, the following logs appear: Now, if we try to do the same with kubernetes 1.23 (planned to ship with openshift 4.10), we see that the above deletion NEVER takes place: steps 1 and 2 from above are the same: the cloud service controller realizes that a node has been labeled (or rebooted) and removes from the list of nodes it passes to the Azure The PR mentioned above, however, changes the logic of how nodes are removed and the result is that deletion never happens, at least for the two cases mentioned above (a node that gets the "exclude" label, a node that is rebooting). From what I understand in the current code: deletion now happens if this call returns true: this function does NOT make use of the node list received as input by the reconciler. It looks for the node name in , which is updated by , which is in turn triggered by separate node informers in for some reason, these node informers trigger a cache update only if the topology label () on them changes, so the list in is only correct at startup, and then becomes stale. As a consequence, in our tests, as nodes reboot or get labeled, the code always checks for inclusion against an empty list... The update logic with these node informers was not part of the PR mentioned above. Is this something that was done on purpose? In order to overcome this, I tried removing the condition on the topology label and triggering a cache update on every update event. This goes a little further, but deletion is skipped because of : Indeed, if I add debug lines I see that: - I am not familiar with these terms and what they entail. What's interesting, though, is that this code hasn't changed all that much since openshift 4.9, so I debug lines to see what the values for and were back then and it turned out that they were both empty strings! The check for equality was then successful and deletion took place. Now, why were they both empty?

is an input parameter in and is set here, in the reconciler function: In one of my tests, held the same value as (), so in the code below we call : It turns out, returns an empty value in openshift 4.9: This code wasn't modified in the PR or since 1.22, so I do not see how now with 1.23 we finally get a proper value for ... So to recap, The node cache updates take place only when the topology label changes. Was this done on purpose? I don't see it mentioned in the issue linked to the original PReven if we trigger updates to the node cache systematically (disregarding the topology label check), we then hit another problem that prevents the delete operation to take place. This involves some subtleties in the Azure configuration (vmasName vs vmSet, and also Config.PrimaryAvailabilitySetName) and we definitely need someone from the upstream community to take a look at any code changes we might introduce in there...
Here I wrote down the steps needed to reproduce this issue, with the necessary files available in the same repo:
(I just updated my long comment above with more precise info on openshift 4.9)
Thanks, & We'll try to get the right folks looking at this.
Thanks , appreciate all your help and efforts on this one!
I am suspecting there is a conflict between: the original PR in the kubernetes-sigs repo ()the code that eventually got merged in kubernetes 1.23 () The update of the internal cache ( occurring only when the topology label changes might not have been in the intention of the original PR author... In the original pull request, the part of runs every time it gets executed, as I suggested in my comment above: In kubernetes 1.23, this got merged with the (pre-existing) checks on the topology labels that are not included in the kubernetes-sigs repo:
thanks for catching this, I will file a PR to fix it ASAP. By the way, is it possible to switch to out-of-tree cloud provider azure since it has been GA?
/assign
/triage accepted
Let me see if I got this right.

In you say the following: Before was merged, a node was removed from the backend pool of a load balancer if: status was no longer Ready or was labeled with This decision happened in the cloud provider controller: ... and the result of that was passed over to the azure-specific code in that you quote in the PR: Now, with this PR merged, a node gets removed from the backend pool of a load balancer only if: is labeled with or is labeled with ( ) This means that nodes that are no longer Ready still appear in the cloud loadbalancer backend pools, whose contents no longer reflect available nodes. Is this really the desired behaviour? Were there good reasons behind this decision? Could we add a couple of lines in in that also lookup the status of a node? What are your thoughts on this? Thanks a lot!
The fix would be rather simple, something along the lines of What do you guys think? Thanks!


Question: } } else { // Volume is attached to node, detach it <ins> // Kubelet not responsible for detaching or this volume has a non-attachable volume plugin. </ins> if rc.controllerAttachDetachEnabled || !attachedVolume.PluginIsAttachable { <del> // Kubelet not responsible for detaching or this volume has a non-attachable volume plugin, // so just remove it to actualStateOfWorld without attach. rc.actualStateOfWorld.MarkVolumeAsDetached( attachedVolume.VolumeName, rc.nodeName) </del> <ins> rc.actualStateOfWorld.MarkVolumeAsDetached(attachedVolume.VolumeName, attachedVolume.NodeName) glog.Infof("Detached volume %q (spec.Name: %q) devicePath: %q", attachedVolume.VolumeName, attachedVolume.VolumeSpec.Name(), attachedVolume.DevicePath) </ins> } else { // Only detach if kubelet detach is enabled glog.V(12).Infof("Attempting to start DetachVolume for volume %q (spec.Name: %q)",

During testing a AWS cluster, found AWS volume once was pending for detach about 30 minutes. According to the log, the following happened, controller issues a detach operation after detach response returns, it periodically checks the status until it reaches "detached". However, in this test, the status is "busy" all the time. after reaching the timeout 30 mins, detach function returns error because the status is not "detached" The following is the log related to the failed test I0915 18:29:46. 5 ] Verified volume is safe to detach for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-". E0915 18:59:54. 5 ] Error detaching volumeID "vol-": Timeout waiting for volume state: actual=busy, desired=detached E0915 18:59:54. 5 ] Operation for """" failed. No retries permitted until 2016-09-15 18:59:55. +0000 UTC (durationBeforeRetry 500ms). Error: DetachVolume.Detach failed for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-" with: Timeout waiting for volume state: actual=busy, desired=detached
Thoughts on what the ideal timeout should be. Too short would be bad because it would give a false-negative.
I know this is from a test but I'm seeing the same problem on a production cluster (1.4.8). I'm not sure exactly what triggers it yet. Kubelet only unmounts the pod directory, not the mount in "aws-ebs/mounts". As a result, the EBS gets stuck in "busy" state when detaching since it is still mounted to the VM. Manually unmounting the EBS allows it to get unstuck and detach safely. In controller manager I see: It then proceeds to get stuck detaching (volume goes to busy state), after which it starts trying to force detach which of course hangs indefinitely.

Inspecting the node directly, I see it's still mounted which is why detach hangs: So something in kubelet is messed up and not unmounting both mount points correctly.
I think this might be the same issue as
On kubelet, it looks like the volume failed to mark as mounted, which is why it subsequently got stuck. This may have to do with the fact this pod () got rolled back while waiting for the previous deployment to succeed.
^ Looks like it might be a race condition between the controller manager and reconciler: manager attaches volume starts attachment process (pod rollback/delete) controller manager thinks its safe to detach, so removes volume from node's volume list, then tries to detach mounts the volume as part of the ongoing attach/mount process from (2) tries to update the mount list, this fails because something (reconciler?) has removed the volume from attached list gets stuck in (busy) state detaching, as its mounted to the kubelet node. it never gets unmounted again in kubelet because state doesn't reflect it is mounted.
Looks like there exists race conditions between external updates of and 's asynchronous kick off of operations like MountVolume, which is the case I'm seeing. Seems like a lot of these edge cases could crop up here - maybe reconciler should lock changes to the desiredStateOfWorld while it's running?
thank you for providing detailed information and analysis. I think there is a race condition between asynchronous operation MountVolume and updating actualStateOfWorld directly by reconciler I will work on a fix. Please let me know if you have any issue/comments.


Question: } dryRun := cmdutil.GetFlagBool(cmd, "dry-run") <ins> output := cmdutil.GetFlagString(cmd, "output") shortOutput := output == "name" </ins> encoder := f.JSONEncoder() decoder := f.Decoder(false)

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: "${e2e}" "${auth_config[@]:+${auth_config[@]}}"  --host="https://${KUBE_MASTER_IP-}"  --provider="${KUBERNETES_PROVIDER}"  <del> ${E2E_REPORT_DIR+"--report_dir=${E2E_REPORT_DIR}"} </del> <ins> ${E2E_REPORT_DIR+"--report_dir=${E2E_REPORT_DIR}"}  "${@}" </ins>

Twice in the last two weeks I have dealt with "bugs" which turned out to be due to the fact that the fundamental requirements of pod networking were not being met. We need a way for a user to very quickly verify that the basic assumptions are met: pods can talk to pods, master can talk to pods, pods can talk to master, services work, etc.
Sounds like something that should have a dual life as an e2e test.
Yes, I ran into the similar issues when I tried to have containervm image as the default one. It turns out just some simple network configuration issues.
Service and replication controller that creates two pods, where the pod takes the kube service as an argument and tries to connect to the other pods described by endpoints?
Yeah that's sorta what I was thinking, too.
After gets merged, I think the easiest path to finishing this off is to just parameterize so that it's easy to point it at an existing cluster and run only the network test.
This is fixed by
Not completely fixed yet; need simple way to run it on existing cluster.
Remaining action item: Write a go script (bash if you must) that runs the networking test from against an existing cluster. Something that you can run via "go run" would be great, or adding flags to such that you can run it with something like "_output/go/bin/e2e -test=network" would be great too.
Reopening. Along the way in , we're moving the e2e script. But I also noticed this is relying on test assets. If you're trying to ship something to users, isn't something we ship to users.
Running will work after . We might end up renaming the test as part of the Ginkgo migration, in which case we can update this issue to indicate the correct invocation.
Closing this as is merged.


Question: func TestAppendDefaultMountOptions(t *testing.T) { tests := []struct { options  []string <ins> fsGroup  *int64 </ins> expected []string }{ { <del> options:  []string{"dir_mode=0777"}, expected: []string{"dir_mode=0777", fmt.Sprintf("%s=%s", fileMode, defaultFileMode), fmt.Sprintf("%s=%s", vers, defaultVers)}, </del> <ins> options: []string{"dir_mode=0777"}, fsGroup: nil, expected: []string{"dir_mode=0777", fmt.Sprintf("%s=%s", fileMode, defaultFileMode), fmt.Sprintf("%s=%s", vers, defaultVers)}, </ins> }, { <del> options:  []string{"file_mode=0777"}, expected: []string{"file_mode=0777", fmt.Sprintf("%s=%s", dirMode, defaultDirMode), fmt.Sprintf("%s=%s", vers, defaultVers)}, </del> <ins> options: []string{"file_mode=0777"}, fsGroup: to.Int64Ptr(0), expected: []string{"file_mode=0777", fmt.Sprintf("%s=%s", dirMode, defaultDirMode), fmt.Sprintf("%s=%s", vers, defaultVers), fmt.Sprintf("%s=0", gid)}, </ins> }, { <del> options:  []string{"vers=2.1"}, expected: []string{"vers=2.1", fmt.Sprintf("%s=%s", fileMode, defaultFileMode), fmt.Sprintf("%s=%s", dirMode, defaultDirMode)}, </del> <ins> options: []string{"vers=2.1"}, fsGroup: to.Int64Ptr(1000), expected: []string{"vers=2.1", fmt.Sprintf("%s=%s", fileMode, defaultFileMode), fmt.Sprintf("%s=%s", dirMode, defaultDirMode), fmt.Sprintf("%s=1000", gid)}, </ins> }, { <del> options:  []string{""}, expected: []string{"", fmt.Sprintf("%s=%s", fileMode, defaultFileMode), fmt.Sprintf("%s=%s", dirMode, defaultDirMode), fmt.Sprintf("%s=%s", vers, defaultVers)}, </del> <ins> options: []string{""}, expected: []string{"", fmt.Sprintf("%s=%s", fileMode, defaultFileMode), fmt.Sprintf("%s=%s", dirMode, defaultDirMode), fmt.Sprintf("%s=%s", vers, defaultVers)}, </ins> }, { options:  []string{"file_mode=0777", "dir_mode=0777"}, expected: []string{"file_mode=0777", "dir_mode=0777", fmt.Sprintf("%s=%s", vers, defaultVers)}, }, <ins> { options: []string{"gid=2000"}, fsGroup: to.Int64Ptr(1000), expected: []string{"gid=2000", fmt.Sprintf("%s=%s", fileMode, defaultFileMode), fmt.Sprintf("%s=%s", dirMode, defaultDirMode), "vers=3.0"}, }, </ins> } for _, test := range tests { <del> result := appendDefaultMountOptions(test.options) </del> <ins> result := appendDefaultMountOptions(test.options, test.fsGroup) </ins> if !reflect.DeepEqual(result, test.expected) { t.Errorf("input: %q, appendDefaultMountOptions result: %q, expected: %q", test.options, result, test.expected) }

If I look at the code of azureFile plugin, it has fsGroup as an argument, but don't see its usage anywhere I also don't see anywhere explicitely stated that azureFile volume supports fsGroup.
/sig storage
For those landing here, a separate path to get the mount point to have the right permissions is to add this annotation to the PV metadata: This makes the plugin mount the volume in the Pod under those UID,GID. Obviously, this implies that one knows what those are for their workload in the container ...
If i create a PV like this, still mounted with the default root used into POD. StorageClass PV
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc Let's check whether works for azure file.
/sig azure
is another way to set groupid(gid) While according to this , we also need to support Here is the code about setting fsgroup from securityContext. I will enable this feature and do a test, it should be 3 lines of code. /assign
BTW, this is all for Linux(UserID, GroupID), is there an identical setting for Windows container?
No, setting user for volumes doesn't work on Windows. Only the rw/ro options are implemented.
According to my investigation, though I put the following code in azure file , it does not take effect. The reason is azure file(cifs) mount does not support gid setting after file system mount, by default would be 0(root) So in summary, set fsGroup code in azure file is useless, user could set gid by mountOptions in storage class, here is an Let me know if you hav any concern about this issue Update: Root cause is azure file(cifs) mount does not support gid setting after file system mount, while azure disk supports.
is this true, that cifs does not support guid setting?
Of course cifs support gid setting, see example: I mean cifs does not support second time gid setting, kind of differnet than azure disk.

Anyway, I am working on a PR to support if no gid setting in azure file storage class, while if both sets, only gid from storage class would be used.
while the code is quite different than just add below 3 lines of code, actually below does not work: I will show you the PR tomorrow.


Question: if err := Convert_v1_Context_To_api_Context(&curr.Context, newContext, s); err != nil { return err } <ins> if *out == nil { *out = make(map[string]*api.Context) } </ins> if (*out)[curr.Name] == nil { (*out)[curr.Name] = newContext } else {

What happened: I have a test case: This behaves different with different client-go versions. With , the test passes. This is using in with in With , this test panics. This is using in with in Panic logs: Basically it its creating an object to decode into, and this object has some maps. These are set to nil, so when we try to populate them we get a panic. What you expected to happen: Behavior remains the same as previous release, or at least have some alternative that works here? How to reproduce it (as minimally and precisely as possible): I pasted the test above that reproduces this. This can be reproduced with a project with just that file, and a file with only client-go. Using go 1.13
/sig api-machinery
/assign


Question: fi replace_prefixed_line "${basic_auth_csv}" "${KUBE_PASSWORD},${KUBE_USER}," "admin,system:masters" fi local -r known_tokens_csv="${auth_dir}/known_tokens.csv" <ins> if [[ -e "${known_tokens_csv}" && "${METADATA_CLOBBERS_CONFIG:-false}" == "true" ]]; then rm "${known_tokens_csv}" fi </ins> if [[ -n "${KUBE_BEARER_TOKEN:-}" ]]; then replace_prefixed_line "${known_tokens_csv}" "${KUBE_BEARER_TOKEN},"             "admin,admin,system:masters" fi

/sig cluster-lifecycle
Very rarely we are seeing known tokens files filled with the NULL character, with ASCII value 000. Restarting the master doesn't fix the issue because the is saved between restarts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: network_mode="custom" fi echo "Creating new ${network_mode} network: ${NETWORK}" <del> gcloud compute networks create --project "${NETWORK_PROJECT}" "${NETWORK}" --mode="${network_mode}" </del> <ins> gcloud compute networks create --project "${NETWORK_PROJECT}" "${NETWORK}" --subnet-mode="${network_mode}" </ins> else PREEXISTING_NETWORK=true PREEXISTING_NETWORK_MODE="$(check-network-mode)"

/kind bug What happened: Warning printed by : What you expected to happen: No warning. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This is my first e2e cluster. Logs with more context below. Environment: Kubernetes version (use ): master Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): Description: Ubuntu 14.04 LTS
This is probably an issue in the scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
This has started failing in tests: https://k8s-
/shrug /milestone v1.10 /assign /assign /priority failing-test /priority critical-urgent /sig network /sig testing
I think this might be only in the staging gcloud release; at least doesn't mention this being removed.
and yeah, gce-canary is also using staging gcloud /shrug
/assign I'll fix kubetest, is patching k8s
/approved-for-milestone
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release.

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/status in-review
should we close this issue since the PR merged?
waiting for , and that need to be cherrypicked into older releases
still need to cherrypick into previous releases
agreed, but isn't blocking 1.10 any more /milestone clear
cherrypick is done, closing /close


Question: if err != nil { return err } <del> if !labels.AreLabelsInWhiteList(pod.Spec.NodeSelector, whitelist) { </del> <ins> if !isSubset(pod.Spec.NodeSelector, whitelist) { </ins> return errors.NewForbidden(resource, pod.Name, fmt.Errorf("pod node label selector labels conflict with its namespace whitelist")) }

What happened: The function return when whitelist is empty. The function is implemented as follows: What you expected to happen: When the whitelist is empty, any key of labels can not in it. Should the function return when whitelist is empty?
/sig api-machinery
/assign /cc (we think you might have opinions on fixing this)
I only see one consumer. Rather than change the semantic, I would suggest moving it to the point of use and renaming it.
Yes, this function is wrong, I agree with David. P.S. we don't advertise interface stability for this library so this is OK. But even if we did, I think callers of this function are probably not expecting this behavior and should be examined.) P.P.S. The way it should work is via set difference, the labels should be a subset of the list. P.P.P.S. These are sets and not lists, and the names should be changed to reflect that. P.P.P.P.S. If you're making a change here anyway, please lose the word "whitelist".
Okay, I will submit a pr.


Question: return ss.(*scaleSet), nil } <del> func buildTestVMSS(lbBackendpoolIDs []string) compute.VirtualMachineScaleSet { </del> <ins> func buildTestVMSS(name string, lbBackendpoolIDs []string) compute.VirtualMachineScaleSet { </ins> lbBackendpools := make([]compute.SubResource, 0) for _, id := range lbBackendpoolIDs { lbBackendpools = append(lbBackendpools, compute.SubResource{ID: to.StringPtr(id)}) } expectedVMSS := compute.VirtualMachineScaleSet{ <del> Name: to.StringPtr(testVMSSName), </del> <ins> Name: &name, </ins> VirtualMachineScaleSetProperties: &compute.VirtualMachineScaleSetProperties{ ProvisioningState: to.StringPtr("Running"), VirtualMachineProfile: &compute.VirtualMachineScaleSetVMProfile{

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestEnsureBackendPoolDeletedConcurrently Testgrid link: Reason for failure: /sig cloud-provider
cc
/assign /area provider/azure


Question: "Metric":    gstruct.Ignore(), "Value":     gomega.BeNumerically(">=", lower), "Timestamp": gstruct.Ignore(), <ins> "Histogram": gstruct.Ignore(), </ins> })) }

There's been a quite a few failures across jobs that query node metrics. The test is Last passing - First failing - appears to point to which is bump in prometheus client library. Possibly related? /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Verified with presubmit that reverting the prometheus client library results in the test passing. Opened PR for revert in
/cc
/assign
thanks for filing this issue! I think seems like a promising fix (thanks ) and there's a fair chance we won't need the revert
ack, thanks for the fix, it seems we can go with that then and hold the revert.


Question: package mount import ( <ins> "errors" </ins> "fmt" <ins> "io/fs" </ins> "os" "strconv" "strings" "syscall" <ins> "k8s.io/klog/v2" </ins> utilio "k8s.io/utils/io" )

What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?

Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,