Question: } } <ins> // Unlink the forwarding chain. args = []string{ "-m", "comment", "--comment", "kubernetes forwarding rules", "-j", string(kubeForwardChain), } if err := ipt.DeleteRule(utiliptables.TableFilter, utiliptables.ChainForward, args...); err != nil { if !utiliptables.IsNotFoundError(err) { glog.Errorf("Error removing pure-iptables proxy rule: %v", err) encounteredError = true } } </ins> // Flush and remove all of our chains. iptablesData := bytes.NewBuffer(nil) if err := ipt.SaveInto(utiliptables.TableNAT, iptablesData); err != nil {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: 3x Ubuntu 16.04 nodes running in VMware vSphere OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic Install tools: kubeadm Others: CNI: Canal (flannel + calico) What happened: I originally opened to make Canal CNI work when the node's FORWARD chain default policy is set to DROP. However, I realized that the DROP policy was the root cause for why my other CNI tests also were failing. When either weave and flannel CNI is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails. Specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop. Also, the kube-dns pod also crashes. <details<summarykubectl get pods --all-namespaces -o wide</summary</details<details<summarykubectl logs --namespace=kube-system kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kube-dns--gz34l</summary</details<details<summarysudo iptables -t filter -v -L FORWARD --line-numbers (197 currently has the dashboard pod scheduled. Note the 2 dropped packets.

)</summary</details</brWhat you expected to happen: Either it should be better documented that the default FORWARD chain policy should be ACCEPT or rules should be to prevent packets from hitting the default policy. As a note, Ubuntu 16.04 defaults the FORWARD chain to DROP. I assume the is why the packets are going through the FORWARD chain in the first place but without , the dashboard still can not access the kube-apiserver. One option that I suggested in is to have kube-proxy add an iptables FORWARD rule to ACCEPT packets that have been marked by KUBE-MARK-MASQ. How to reproduce it (as minimally and precisely as possible): Create a cluster with: Weave or Flannel CNI enabled on all nodes kube-proxy set to kubernetes-dashboard enabled. kubernetes-dashboard will crash often. Anything else do we need to know: is closed because I thought the issue was specific to Canal and instead opened However, this looks to be a larger issue that kubernetes itself should address.
Weave issue:
I think this is going to hit a lot of people as Docker 1.13 is now setting the FORWARD chain to drop by default if ip_forward is set. See
See also
overlap with ?
I think a solution for this issue or (which is closer to what I feel like a proper solution is) should be considered for inclusion for K8s 1.8 since it will officially support docker 1.13. The problem as I see it is that kube-proxy sets up rules to forward traffic for services (specifically I have seen the problem with NodePorts) yet does not ensure or have any requirement (to my knowledge) that the forwarded traffic be allowed when the traffic is forwarded to a host other than the one receiving the traffic. I observed this when using K8s v1.7.5 installed with kubeadm v1.7.5 and Calico v2.5. It can be remedied with the addition of but hopefully there is a better answer than doing that manually and hopefully something more specific to what is needed. I saw the comment on the Docker 1.13 validation issue linking to the configuration script used in the GCE docker images which is why the e2es are not having any issues.
If we can be more precise about which traffic we require to ALLOW, I'm OK with adding rules to open that.

I've been doing some testing with the first two options and chatted with and about these potential options for adding some forwarding rules. They all are a certainly lot more specific than the previous blanket all ALLOW. any traffic sourced or destined for the clusterCIDR. traffic that has the KUBE-MARK-MASQ set and conntrack rules to accept the follow on traffic. a third option of adding a more specific mark that is only set on specific traffic that it is known will need to be forwarded and match on that (this option would probably need the conntrack rules like the above option too) do any of those options appeal to you or do you see any problems with them?
If we add some more rules, we shouldn't put them in the top-level chain since it makes them hard to manage/clean up. Best to add them to a chain that kube-proxy owns that is jumped to from the top-level chain.
doesn't necessarily know the ClusterCIDR. I could see your proposal making sense in conjunction with .
Very good point I should have mentioned that the options I presented are assuming clusterCIDR is set. If clusterCIDR is not defined then the first option is not possible (except for the wide open option that these rules are attempting to avoid). I guess on options 2 & 3 if it was not set then a conntrack rule could be without the clusterCIDR. I would appreciate feedback on the desired behavior if clusterCIDR is not set too.
The PR that was merged solved the issue for pods inside clusterCIDR. But it doesn't handle hostNetwork pods that use the node IP, like when you want to expose the API server as a load-balanced service externally. I'm not sure why the “kubernetes forwarding conntrack pod source rule” has to be limited to the pod CIDR, since --ctstate RELATED,ESTABLISHED should already ensure it does not allow traffic that wasn't already allowed by another rule.
I'd argue that any iptables/firewalling for a pod using isn't the responsibility of Kubernetes, There's no reasonable way for k8s to actually know what interfaces a host-network container binds to, what ports it listens on, or what expectations it has. The cluster operator should probably be responsible for that networking case.
I meant explicitly regarding services where those pods are the endpoints of a service, so Kubernetes does know the address and the port.

Due to limitation of state RELATED,ESTABLISHED in FORWARD rule i.e. current implementation, the unidirectional UDP traffic apps won't work. Because packets (except first) aren't ESTABILISHED from iptables's perspective (no reply from endpoint, no handshake). Shouldn't this be something like and not
To add to comment, bidirectional UDP apps are brittle as well. Dropping or failing to reply to the first packet will prevent the app from ever receiving a second. More discussion on


Question: INSTANCE_PREFIX: $(yaml-quote ${INSTANCE_PREFIX}) NODE_INSTANCE_PREFIX: $(yaml-quote ${NODE_INSTANCE_PREFIX}) NODE_TAGS: $(yaml-quote ${NODE_TAGS:-}) <ins> NODE_NETWORK: $(yaml-quote ${NODE_NETWORK:-}) NODE_SUBNETWORK: $(yaml-quote ${NODE_SUBNETWORK:-}) </ins> CLUSTER_IP_RANGE: $(yaml-quote ${CLUSTER_IP_RANGE:-10.244.0.0/16}) SERVER_BINARY_TAR_URL: $(yaml-quote ${server_binary_tar_url}) SERVER_BINARY_TAR_HASH: $(yaml-quote ${SERVER_BINARY_TAR_HASH})

The "Services should be able to create an internal type load balancer [Slow]" e2e test is failing on our 2k-node gce clusters - https://k8s- with the following error: We're using custom subnets for our large cluster tests, and seems like the glbc controller is not able to handle them. cc
Seems similar to Assigning to you as you seem to have the context already :) Also, is this expected to be fixed in 1.8 timeframe?
doesn't set the necessary variables for the GCE cloud provider. Will throw you a PR.
Cool - thanks a lot!
Ref
I see the last test on 09/09 failed, but I can't see the build log due to exceeding memory. Any other way I can see the log, /reopen
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
nevermind - went directly to raw log.
Friendly ping. This e2e is still failing (https://k8s-) and it's part of release-blocking suite.
/close


Question: found:     testCase.pspFound, }, } <ins> mgr.sourcesReady = &sourcesReadyStub{} </ins> success, failure := mgr.reconcileState() if testCase.expectSucceededContainerName != "" {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: services: []*v1.Service{{Spec: v1.ServiceSpec{Selector: map[string]string{"baz": "blah"}}}}, rss:      []*apps.ReplicaSet{{Spec: apps.ReplicaSetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"foo": "bar"}}}}}, // We use ReplicaSet, instead of ReplicationController. The result should be exactly as above. <del> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 5}}, </del> <ins> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 0}}, </ins> name:         "service with partial pod label matches with service and replica set", }, {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: /kind bug What you expected to happen: Problem 1: Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per (zone or node) resulting in imbalanced cluster. Eg: removes nodes which cannot . If any of the pods are already scheduled on this node, they are not considered when counting in . Problem 2: When there are 2 selectors(service and replication controller), it is sufficient to match any one selector for distribution. This creates imbalance ]. Pods from previous deploys matches and are counted when distributing pods across zones/nodes (Even though they do not match ) . These pods will be deleted. After the deploy completes, the cluster is imbalanced - by zone and/or pods per node. How to reproduce it (as minimally and precisely as possible): Problem 1: Get pods to get scheduled on nodes with very high cpu utilization [Existing cpu utilization + 1 new pod will result in almost equal to 0]. GeneralPredicate will drop this node after scheduling the first pod. This will lead to over utilization of an already loaded zone. If there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone. Problem 2: Have 2 selectors and deploy 30 pods across 3 zones. Twice. Anything else we need to know?: Environment: Kubernetes version (use ): /sig scheduling <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
re Problem 1: for per node, we only need to compare current nodes as they will share the same maxPods. for per zone, we need to account other nodes, e.g. because of resource predicates. re problem 2: a little about your case :(
Problem 1: : It does not matter if the denominator is all nodes or current nodes. One with higher utilization will still end up with a lower score. : It is critical that the denominator is all nodes. Else we are ending up with highly imbalanced cluster. Problem 2: Here is an example list of selectors.

All pods matching any one service selector are counted in So in our case, old pods are counted during scheduling. But they get killed during the deployment. If the maxSurge and maxUnavailable is high enough, you will end up with an imbalanced zone after deployment.
Gentle ping.
/assign bsalamat
Re: Problem 1 - You can also reproduce this by using pod anti-affinity. Zone score is calculated only based on nodes filtered by predicates. IMO zone score should be calculated using all nodes.
I will try to separate this into two PRs. Any other suggestions welcome.
I am working Problem 1. Looks like I have to open a new issue.


Question: // of the pipe. r, w, err := r.os.Pipe() if err != nil { <del> return err </del> <ins> return newRktExitError(err) </ins> } go io.Copy(w, stdin)

With rkt as the container runtime, the e2e test fails with I dove into it a bit further and it looks like a real issue. With additional logging it appears that the probe does fail (exec happens and returns incorrectly as expected), but when calls it never gets that the given ID exists. The pod definition I've been using is: It restarts very slowly with rkt (on the order of an hour I believe, see sleep 600), but restarts very frequently with docker. cc


Question: // equalLoadBalancingRulePropertiesFormat checks whether the provided LoadBalancingRulePropertiesFormat are equal. // Note: only fields used in reconcileLoadBalancer are considered. <del> func equalLoadBalancingRulePropertiesFormat(s, t *network.LoadBalancingRulePropertiesFormat) bool { </del> <ins> func equalLoadBalancingRulePropertiesFormat(s *network.LoadBalancingRulePropertiesFormat, t *network.LoadBalancingRulePropertiesFormat, wantLB bool) bool { </ins> if s == nil || t == nil { return false } <del> return reflect.DeepEqual(s.Protocol, t.Protocol) && </del> <ins> properties := reflect.DeepEqual(s.Protocol, t.Protocol) && </ins> reflect.DeepEqual(s.FrontendIPConfiguration, t.FrontendIPConfiguration) && reflect.DeepEqual(s.BackendAddressPool, t.BackendAddressPool) && reflect.DeepEqual(s.LoadDistribution, t.LoadDistribution) && reflect.DeepEqual(s.FrontendPort, t.FrontendPort) && reflect.DeepEqual(s.BackendPort, t.BackendPort) && <del> reflect.DeepEqual(s.EnableFloatingIP, t.EnableFloatingIP) && reflect.DeepEqual(s.IdleTimeoutInMinutes, t.IdleTimeoutInMinutes) </del> <ins> reflect.DeepEqual(s.EnableFloatingIP, t.EnableFloatingIP) if wantLB { return properties && reflect.DeepEqual(s.IdleTimeoutInMinutes, t.IdleTimeoutInMinutes) } return properties </ins> } // This compares rule's Name, Protocol, SourcePortRange, DestinationPortRange, SourceAddressPrefix, Access, and Direction.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the Idle Timeout annotation was set to "3" which isn't a supported case, kube-controller-manager is stuck in deleting Azure LB. New service with the same name couldn't be created again even without the annotation. What you expected to happen: New service should be created successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug
/assign Since I'm seeing this on some of our clusters
/priority important-soon
Thanks
- The problem here seems that when we want to call (when processing a delete event) it reconciles the loadbalancer which in turn tries to get the idleTimeout, resulting in an error. one of the strategies is to still construct our diff in case of a delete and exercise the code path at the end. This basically translate into ignoring the error when getting idleTimeout and excluding it from finding if the rule is dirty or not when we are . toughts ? Generally speaking, whenever we update a loadbalancer with unsupported values on annotations, ensuring load balancer will fail resulting in an un-cleaned LB, this is a cross-provider issue. /sig cloud-provider cc
cc
lgtm. the delete operation should continue to do cleanup job even some wrong options are provided from users.


Question: // used to create a 1MiB file in the target directory. initCmd := fmt.Sprintf("i=0; while [ $i -lt %d ]; do echo -n %s >>%s; let i+=1; done", loopCnt, writeBlk, ddInput) <del> clientPod := makePodSpec(config, dir, initCmd, volsrc, podSecContext) </del> <ins> clientPod := makePodSpec(config, initCmd, volsrc, podSecContext) </ins> By(fmt.Sprintf("starting %s", clientPod.Name)) podsNamespacer := cs.CoreV1().Pods(config.Namespace)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow] Since when has it been failing: 11/13 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: This is consistently failing in 1.13 branch and is release blocking. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test /sig storage /priority critical-urgent /milestone v1.13 /assign /cc
The test case is failing because it wrote 2MB instead of the expected 1MB. It's also very odd that the test is failing in soak, but passing in regular CI suites.
There's a few problems: .GetVolumeSource returns "/tmp" for everything. This means all tests will share the same directory. This should be probably use some unique directory instead volume IO test generates a unique mountPath directory, however that doesn't mean a unique directory is actually created on the host. Ie, a mountPath of "/a/b/c" on a hostpath volume at "/tmp" doesn't mean directory "/tmp/a/b/c" is created. io test file name is not unique seems like the IO test is not cleaning up the files it creates, so the files are getting reused on subsequent tests due to the non-unique naming issues.
The other thing I can't explain is why it's failing only now
Fix is here:
As per discussion in 1.13 burndown this morning, this is a test job issue and its not release blocking at this point. So we can take this fix in 1.14 as opposed now during code freeze let us know if you feel otherwise
/milestone v1.14 /priority important-soon
/remove-priority critical-urgent


Question: return fmt.Errorf("failed to get node address from cloud provider: %v", err) } if kl.nodeIP != nil { <ins> enforcedNodeAddresses := []v1.NodeAddress{} </ins> for _, nodeAddress := range nodeAddresses { if nodeAddress.Address == kl.nodeIP.String() { <del> node.Status.Addresses = []v1.NodeAddress{ {Type: nodeAddress.Type, Address: nodeAddress.Address}, {Type: v1.NodeHostName, Address: kl.GetHostname()}, } return nil </del> <ins> enforcedNodeAddresses = append(enforcedNodeAddresses, v1.NodeAddress{Type: nodeAddress.Type, Address: nodeAddress.Address}) </ins> } } <ins> if len(enforcedNodeAddresses) > 0 { enforcedNodeAddresses = append(enforcedNodeAddresses, v1.NodeAddress{Type: v1.NodeHostName, Address: kl.GetHostname()}) node.Status.Addresses = enforcedNodeAddresses return nil } </ins> return fmt.Errorf("failed to get node address from cloud provider that matches ip: %v", kl.nodeIP) }

BUG REPORT: /kind bug What happened: After upgrading the kubernetes cluster from 1.6.4 with the VSphere Cloud Provider enabled to 1.6.5/1.6.6 or 1.6.7, the cluster nodes don't have anymore InternalIPs Several issues occur at this step. For example, it breaks heapster with such an error : What you expected to happen: The node should have an InternalIP. How to reproduce it (as minimally and precisely as possible): Deploy a 1 node kubernetes cluster in a VM, and enable the vsphere cloud provider. Anything else we need to know?: with kubernetes 1.6.4 we get : Downgrading to 1.6.4 fixes the issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: It may be related to some changes around PR kubernetes /area platform/vsphere cloudprovider
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-ops /sig network /area platform/vsphere /area cloudprovider
Performing some more tests on a mini instance outside of the production cluster, I think I've identified the issue. The configuration required to start kubelet with the option (otherwise the wrong IP/interface is used to communicate between nodes). The issue is triggered when this option is set. The cloud provider looks OK. But the issue is located here : When the is not , the loop iterates on the node addresses and returns only the first one found. Since PR kubernetes, the list always returns an ExternalIP first. I think the loop in should iterate on all the addresses to rebuild the list, so that it will include the ExternalIP and the InternalIP. Can someone confirm the analysis ?
/sig node
ping
Modifying the code like this solved the issue : which returns : I guess it can be done in a cleaner way, but it was to quickly test the fix.

CC:
Thanks for filing this issue related to vSphere cloud provider. I agree with your analysis. It seems the current function assumes that the IP address specified by user can only be InternalIP. Your fix to is a better solution than work-around in cloud provider.
Hi, what are the next steps to discuss about the PR ? Are you able to make some tests ?


Question: } } <ins> func runResourceTrackingTest(framework *Framework, podsPerNode int, nodeNames sets.String, resourceMonitor *resourceMonitor) { numNodes := nodeNames.Len() totalPods := podsPerNode * numNodes By(fmt.Sprintf("Creating a RC of %d pods and wait until all pods of this RC are running", totalPods)) rcName := fmt.Sprintf("resource%d-%s", totalPods, string(util.NewUUID())) // TODO: Use a more realistic workload Expect(RunRC(RCConfig{ Client:    framework.Client, Name:      rcName, Namespace: framework.Namespace.Name, Image:     "gcr.io/google_containers/pause:go", Replicas:  totalPods, })).NotTo(HaveOccurred()) // Log once and flush the stats. resourceMonitor.LogLatest() resourceMonitor.Reset() By("Start monitoring resource usage") // Periodically dump the cpu summary until the deadline is met. // Note that without calling resourceMonitor.Reset(), the stats // would occupy increasingly more memory. This should be fine // for the current test duration, but we should reclaim the // entries if we plan to monitor longer (e.g., 8 hours). deadline := time.Now().Add(monitoringTime) for time.Now().Before(deadline) { Logf("Still running...%v left", deadline.Sub(time.Now())) time.Sleep(reportingPeriod) timeLeft := deadline.Sub(time.Now()) Logf("Still running...%v left", timeLeft) if timeLeft < reportingPeriod { time.Sleep(timeLeft) } else { time.Sleep(reportingPeriod) } logPodsOnNodes(framework.Client, nodeNames.List()) } By("Reporting overall resource usage") logPodsOnNodes(framework.Client, nodeNames.List()) resourceMonitor.LogCPUSummary() resourceMonitor.LogLatest() By("Deleting the RC") DeleteRC(framework.Client, framework.Namespace.Name, rcName) } </ins> var _ = Describe("Kubelet", func() { <del> var numNodes int </del> var nodeNames sets.String framework := NewFramework("kubelet-perf") var resourceMonitor *resourceMonitor

For example: I'm disabling the test in But we need to re-enable it, or roll-back the test.
fyi
Shouldn't this is part slow test, and the fix should be either: 1) Increase Kubernetes-e2-gke-ci timeout value if decide to include slow to that project or 2) disable all slow tests on that project? cc/ Which one is required here?
The test runs 50 pods per node, which is over the default --max-pod limit of kubelet. The test relies on the flag --max-pods=100 being set in cluster/gce/config- I will disable it for all GKE tests in On the other hand, it's not clear to me where the timeout is set for a specific build. /cc
Timeouts for each Jenkins build are in the Jenkins project configurations themselves, which are not currently saved in version control anywhere. :
Discussed with about the test offline. There are several issues: Do we want to run any kind of kubelet resource monitoring test on GKE? Is it useful? We only override max-pods in GCE. Should the test itself figure out what platform it can run based on the #pods in the test (see ) Per-test configuration allows greater flexibility (e.g., we may run 100 pods on GCE to test kubelet's scalability), but it'd make the tests more complicated as they need to be aware of the test environment (which many e2e tests already do); there is also the burden of updating the tests once the max-pod value changes. Does this test belong to SLOW tests or should it belong to a new category? Not all slow tests have the same attributes. Arguably this test could be enabled only on selective builds. The downside would be more flags -- less readable/manageable e2e configs. Not all builds which run SLOW tests have the proper timeout value to run this test. We need to manually change them every time a slow test is . Re-assigning this back to since she has more concerns about the PR.
Had offline discussion with We are going to split current Kubelet resource usage tracking test into two, one is targeted for GCE, another one is for GKE. In this case, we won't mess up the test code itself by introducing knobs. Re-assign this back to


Question: // TODO(liggitt): drop this once golang json parser limits stack depth (https://github.com/golang/go/issues/31789) if len(p.patchBytes) > 1024*1024 { v := map[string]interface{}{} <del> if err := json.Unmarshal(p.patchBytes, v); err != nil { </del> <ins> if err := json.Unmarshal(p.patchBytes, &v); err != nil { </ins> return nil, errors.NewBadRequest(fmt.Sprintf("error decoding patch: %v", err)) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It returns an error when I create a pod of yaml file over 1M. I checked the codes and found that the coded wrong. It should be “json.Unmarshal(p.patchBytes, &v)”, not “json.Unmarshal(p.patchBytes, v)” in the 344 and 368 line. What you expected to happen: It can be parsed correctly. How to reproduce it (as minimally and precisely as possible): Look at the code and you'll see or you can test it. Anything else we need to know?: no. Environment: Kubernetes version (use ): Community latest code Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind bug
/sig api-machinery
/assign


Question: // Waiting for all controller sync. internalInformers.Start(controllerCh) internalInformers.WaitForCacheSync(controllerCh) <ins> informers.Start(controllerCh) informers.WaitForCacheSync(controllerCh) </ins> // ------------------------------------------- // Test TaintNodeByCondition feature.

While lookint into , I spent few minutes looking into this test: And there is a problem that is relatively easy to reproduce. The problem is that we are running its test cases one by one. But nothing in the meantime is ensuring that the current state that the test assumes is reflected in scheduler. In particular the problem that I observed couple time is that: we finished one test case we call cleanupNodes [but we don't ensure that scheduler actually cleaned up this node] we start the next test case we try to schedule a pod that has some toleration (but shouldn't fit on our node) by accident the toleration make it fit on to the previous node so scheduler schedules it on the node from previous test case
/assign


Question: cacher.ready.wait() // Ensure there is some budget for slowing down processing. <del> // When using the official `timeBudgetImpl` we were observing flakiness // due under the following conditions: </del> <ins> // We use the fakeTimeBudget to prevent this test from flaking under // the following conditions: </ins> // 1) the watch w1 is blocked, so we were consuming the whole budget once //    its buffer was filled in (10 items) // 2) the budget is refreshed once per second, so it basically wasn't

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: Reason for failure: Anything else we need to know: /assign
/sig api-machinery /triage accepted
OK - I can quite reliably reproduce this problem by adding before this line: This seems to be a test issue (watch being closed as a result of not being able to process stuff on time). I will think how fix the test and make it not suffer from heavy cpu starvation.
Opened fix in


Question: // TODO: move thic logic into cni bridge plugin and remove this from kubenet func (plugin *kubenetNetworkPlugin) ensureMasqRule() error { <del> if _, err := plugin.iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, utiliptables.ChainPostrouting, "-m", "comment", "--comment", "kubenet: SNAT for outbound traffic from cluster", "-m", "addrtype", "!", "--dst-type", "LOCAL", "!", "-d", plugin.nonMasqueradeCIDR, "-j", "MASQUERADE"); err != nil { return fmt.Errorf("Failed to ensure that %s chain %s jumps to MASQUERADE: %v", utiliptables.TableNAT, utiliptables.ChainPostrouting, err) </del> <ins> if plugin.nonMasqueradeCIDR != "0.0.0.0/0" { if _, err := plugin.iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, utiliptables.ChainPostrouting, "-m", "comment", "--comment", "kubenet: SNAT for outbound traffic from cluster", "-m", "addrtype", "!", "--dst-type", "LOCAL", "!", "-d", plugin.nonMasqueradeCIDR, "-j", "MASQUERADE"); err != nil { return fmt.Errorf("Failed to ensure that %s chain %s jumps to MASQUERADE: %v", utiliptables.TableNAT, utiliptables.ChainPostrouting, err) } </ins> } return nil }

When the ip-masq-agent is activated, kubelet installs a rule that looks like: which is nonsensical.


Question: reportedLevels:      []uint64{100, 300, 500, 250, 250}, reportedCPURequests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, useMetricsAPI:       true, <ins> recommendations:     []timestampedRecommendation{}, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ <del> {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "ScaleDownStabilized"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionTrue, Reason: "ValidMetricFound"}, {Type: autoscalingv1.ScalingLimited, Status: v1.ConditionFalse, Reason: "DesiredWithinRange"}, </del> <ins> {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "InvalidMetricSourceType"}, </ins> }, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If you have 2 or more metrics defined on an HPA and one metric is suggesting to scale down and the other metric is unavailable, the HPA will scale down despite missing metrics. Example: Imagine the following Horizontal Pod Autoscaler resource: And the deployment is currently scaled to replicas. If the metric is suggesting to scale to 1 AND the metric is unavailable e.g. HPA can't fetch the metrics. Then it will happily scale down to despite not knowing what the currently unavailable metric would suggest. It's clearly stated in the docs the scale down should not happen if one or more metrics are unavailable: Until Kubernetes v1.15.x the behavior was as expected. However with the . The logic was removing the special handling of stopping scale down in case of unavailable metrics. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /bug /sig autoscaling
/triage accepted


Question: SRCS=check_metadata_concealment ARCH ?= amd64 TARGET ?= $(CURDIR) <ins> GOARM ?= 7 </ins> GOLANG_VERSION ?= latest SRC_DIR = $(notdir $(shell pwd)) export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Building the image binaries requires the GOARM variable to be set. Because it is not defined in all Makefiles, cannot the image binaries for the images in question. What you expected to happen: should succeed. How to reproduce it (as minimally and precisely as possible): git clone cd kubernetes/test/images/entrypoint-tester make Anything else we need to know?: Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: return nil, err } if resp.StatusCode != http.StatusOK { <del> if _, _, err := r.transformResponse(resp, req, nil); err != nil { return nil, err </del> <ins> if result := r.transformResponse(resp, req); result.err != nil { return nil, result.err </ins> } return nil, fmt.Errorf("for request '%+v', got status: %v", req.URL, resp.StatusCode) }

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: } } <ins> func TestGetPublicIPAddressResourceGroup(t *testing.T) { az := getTestCloud() for i, c := range []struct { desc        string annotations map[string]string expected    string }{ { desc:     "no annotation", expected: "rg", }, { desc:        "annoation with empty string resource group", annotations: map[string]string{ServiceAnnotationLoadBalancerResourceGroup: ""}, expected:    "rg", }, { desc:        "annoation with non-empty resource group ", annotations: map[string]string{ServiceAnnotationLoadBalancerResourceGroup: "rg2"}, expected:    "rg2", }, } { t.Run(c.desc, func(t *testing.T) { s := &v1.Service{} s.Annotations = c.annotations real := az.getPublicIPAddressResourceGroup(s) assert.Equal(t, c.expected, real, "TestCase[%d]: %s", i, c.desc) }) } } </ins> func TestGetServiceTags(t *testing.T) { tests := []struct { desc     string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: LoadBalancer is stuck on pending state when annotation azure-load-balancer-resource-group is set to empty string. The error would be: Though the error message introduces some confusion here, it's actually because the resourceGroup name is missing when cloud provider trying to list publicIPs: What you expected to happen: The service should be provisioned successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.8 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /kind bug /priority critical-urgent
/assign


Question: if !configEnabled { framework.Skipf("unable to run test without dynamic kubelet config enabled.") } <ins> // because adminssion Priority enable, If the priority class is not found, the Pod is rejected. systemCriticalPriority := &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: systemCriticalPriorityName}, Value: scheduling.SystemCriticalPriority} </ins> // Define test pods nonCriticalGuaranteed := getTestPod(false, guaranteedPodName, v1.ResourceRequirements{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: assert.EqualValues(t, p0Time.Unix(), ps.StartTime.Time.Unix()) checkNetworkStats(t, "Pod0", seedPod0Infra, ps.Network) <del> checkEphemeralStats(t, "Pod0", []int{seedPod0Container0, seedPod0Container1}, []int{seedEphemeralVolume1, seedEphemeralVolume2}, ps.EphemeralStorage) </del> <ins> checkEphemeralStats(t, "Pod0", []int{seedPod0Container0, seedPod0Container1}, []int{seedEphemeralVolume1, seedEphemeralVolume2}, nil, ps.EphemeralStorage) </ins> if ps.CPU != nil { checkCPUStats(t, "Pod0", seedPod0Infra, ps.CPU) }

Rotated container logs space usage doesnt seem to be counted towards the epehemral storage limit. Always only current log file stats are taken into account. rotated log files usage also should be counted towards ephemeral-storage usage No response <detailsTested on Kubernetes Version : 1.21.7, 1.22 Container Runtime : Cri-o </details<details</details<details</details<details</details<details</details<details</details
/sig node
/cc
/assign we can see if there is a issue
Were the rotated log files in /var/log/pods generated by yourself(not by k8s)? The cadvisor can only collect log files on container's filesystem.
No, I have not created any files. they are generated by k8s itself.
Okay, I see. As the cri-o runtime uses cadvisor stats provider to get container stats, it caculates container logUsage by formular . but the default root dir or log dir of cri-o is not /var/log/pods/{podName}/, so theses log files are not on container's filesystem, then cadvisor may not caculate the data accurately. /cc the containerd uses cri stats provider and gets logstats by HostStatsProvider, it can get right data.
In the kubelet code, , the log path is in the format , but the directory of your environment is , I did not see , Did you set this path yourself?
sorry I truncated the name for better readability, the actual path name is . So I suppose its taking in the above format only.
log files gets generated in the path /var/log/pods/defaultephem15334a37d-df02-435c-b385-/con2 for cri-o in my setup
/area cadvisor /triage accepted /priority important-longterm
/retitle Rotated container log files size not counted towards ephemeral-storage's limit on CRI-O
hi, In the past two days, I have studied the collection of file information by cAdvisor. docker and cri-o runtime basically have the same mechanism in cAdvisor. I have verified that when the runtime is cri-o, the log information statistics are also correct. So I need you to provide more detailed information. I found that the command you show is: , please help to confirm whether the "logs" information belongs to the container con2.

or use the command to try
I have verified and info provided is for con2 only . I have the sample yaml used for testing , please let me know if I'm doing it right or anything missing. I have set the kubelet log configuration as containerLogMaxSize: "5Ki", containerLogMaxFiles: 5 After applying the above yaml stats summary shows con1 logs usedbytes always 0 , con2 logs usedbytes as 4096
attached the stats summary of the command And I'm testing on kubernetes 1.22 release
/assign
what version of CRI-O/kubernetes are you using? using latest main of both kubernetes and cri-o with your reproducer causes the container to be evicted because of it exceeding ephemeral storage
oh I see now: my mistake
another mistake: I was using CRI stats provider, instead of cAdvisor stats provider (as is default)
Is CRI stats provider used by default in 1.23 by default? And also problem is seen only if containerLogMaxSize is less that the ephemeral storage limit set , if its more than the ephemeral storage limit , its the right behavior , pod gets evicted in my setup.
at long last, I believe I've found the issue!


Question: // InitHostPathV0CSIDriver returns a variant of hostpathCSIDriver with different manifests. func InitHostPathV0CSIDriver() testsuites.TestDriver { return initHostPathCSIDriver("csi-hostpath-v0", <del> map[testsuites.Capability]bool{testsuites.CapPersistence: true, testsuites.CapMultiPODs: true}, </del> <ins> map[testsuites.Capability]bool{testsuites.CapPersistence: true, testsuites.CapMultiPODs: true, testsuites.CapSingleNodeVolume: true}, </ins> nil, /* no volume attributes -> no ephemeral volume testing */ // Using the current set of rbac.yaml files is problematic here because they don't // match the version of the rules that were written for the releases of external-attacher

What happened: When we run csi hostpath driver e2e tests on a multi-node cluster some tests failed because they assume that volumes are node independent but this is not the case for some drivers like hostpath. Related PR; What you expected to happen: Skip the tests that assume volumes are node independent for the drivers that don't support node independent volumes. /sig storage /assign


Question: "//vendor/github.com/onsi/ginkgo:go_default_library", "//vendor/github.com/onsi/gomega:go_default_library", "//vendor/k8s.io/api/apps/v1beta1:go_default_library", <ins> "//vendor/k8s.io/api/apps/v1beta2:go_default_library", </ins> "//vendor/k8s.io/api/batch/v1:go_default_library", "//vendor/k8s.io/api/batch/v2alpha1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library",

What happened: There are no e2e and integration tests for statefulset scale subresource. What you expected to happen: (1) e2e test should exist BEFORE code freeze (2) integration test should exist AFTER 1.8 release branches cut Anything else we need to know?: Refer PR for more info. We should NOT remove existing e2e test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close


Question: } klog.V(2).Infof("ensurePublicIPExists for service(%s): pip(%s) - creating", serviceName, *pip.Name) } <del> if len(domainNameLabel) == 0 { pip.PublicIPAddressPropertiesFormat.DNSSettings = nil } else { pip.PublicIPAddressPropertiesFormat.DNSSettings = &network.PublicIPAddressDNSSettings{ DomainNameLabel: &domainNameLabel, </del> <ins> if foundDNSLabelAnnotation { if len(domainNameLabel) == 0 { pip.PublicIPAddressPropertiesFormat.DNSSettings = nil } else { pip.PublicIPAddressPropertiesFormat.DNSSettings = &network.PublicIPAddressDNSSettings{ DomainNameLabel: &domainNameLabel, } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: } } } <ins> func TestDismissLocalhostRuleExist(t *testing.T) { tests := []struct { name     string ipFamily v1.IPFamily src      string }{ { name:     "ipv4 rule", ipFamily: v1.IPv4Protocol, src:      "127.0.0.0/8", }, { name:     "ipv6 rule", ipFamily: v1.IPv6Protocol, src:      "::1/128", }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { ipt := iptablestest.NewFake() if test.ipFamily == v1.IPv6Protocol { ipt = iptablestest.NewIPv6Fake() } ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, test.ipFamily) fp.syncProxyRules() rules := getRules(ipt, kubeServicesChain) if len(rules) <= 0 { t.Errorf("skip loop back ip in kubeservice chain not exist") return } if !rules[0].Jump.Matches("RETURN") || !rules[0].SourceAddress.Matches(test.src) { t.Errorf("rules not match, expect jump: %s, got: %s; expect source address: %s, got: %s", "RETURN", rules[0].Jump.String(), test.src, rules[0].SourceAddress.String()) } }) } } </ins>

Loopback addresses (127.0.0.0/8, ::1/128) can't be used by services in proxy-mode=ipvs it makes sense to dismiss those early, add RETURN rule to KUBE-NODE-PORT,KUBE-SERVICES No response <details</details25 <details</detailsnone <details</details<details</details<details</details<details</details
/sig network
As mentioned in , this is not a bug since the nodeport and ephemeral ranges overlap.
/kind cleanup /kind remove-bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /kind cleanup /kind remove-bug </details
/remove-kind bug
The title of this issue should be changed to something like; Which would indicate that this is not a bug fix but an improvement. Also remove the invalid use-case in the description and add references to comments in like I think to dismiss nodePort connects to localhost early for proxy-mode=ipvs is an improvement, and in the majority of systems traffic to localhost is more common than traffic to nodePorts so this change will also improve overall performance. This is not a fix for overlapping ephemeral and nodePort ranges! So if the title and description is not updated the correct triage is to close this issue as it violates the non overlapping rule for ephemeral/nodeport ranges, please see
/area ipvs
Ok, i will change issue title and descripiton
/triage accepted


Question: case "": default: modes := []string{string(kubeproxyconfig.ProxyModeUserspace), string(kubeproxyconfig.ProxyModeIPTables), string(kubeproxyconfig.ProxyModeIPVS)} <del> errMsg := fmt.Sprintf("must be %s or blank (blank means the best-available proxy (currently iptables)", strings.Join(modes, ",")) </del> <ins> errMsg := fmt.Sprintf("must be %s or blank (blank means the best-available proxy [currently iptables])", strings.Join(modes, ",")) allErrs = append(allErrs, field.Invalid(fldPath.Child("ProxyMode"), string(mode), errMsg)) } return allErrs } func validateProxyModeWindows(mode kubeproxyconfig.ProxyMode, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{} switch mode { case kubeproxyconfig.ProxyModeUserspace: case kubeproxyconfig.ProxyModeKernelspace: default: modes := []string{string(kubeproxyconfig.ProxyModeUserspace), string(kubeproxyconfig.ProxyModeKernelspace)} errMsg := fmt.Sprintf("must be %s or blank (blank means the most-available proxy [currently userspace])", strings.Join(modes, ",")) </ins> allErrs = append(allErrs, field.Invalid(fldPath.Child("ProxyMode"), string(mode), errMsg)) } return allErrs

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When running , and passing the flag which was introduced in , the following error occurs: What you expected to happen: The proxy should work as it used to, supporting this mode. How to reproduce it (as minimally and precisely as possible): ./kube- --proxy-mode=kernelspace Anything else we need to know?: This was introduced after the validation was refactored in which regressed validation for . IPVS validation (and thus support) was later , but we need another mode for Windows. Environment: Kubernetes version (use ): v1.10.0-alpha.0.684+ Cloud provider or hardware configuration: n/a OS (e.g. from ): Windows Server Core 1709 Kernel (e.g. ): Install tools: Others:
/sig windows /sig network
Tentatively adding milestone so we can track getting consensus on release-blocking status.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: software-properties-common  lsb-release <ins> # focal repo for docker is not yet available, so we use bonic for now # https://github.com/kubernetes/kubernetes/issues/90709 release=$(lsb_release -cs) if [ "$release" == "focal" ]; then release="bionic"; fi </ins> # Add the Docker apt-repository curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg  | apt-key add - add-apt-repository  "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")  <del> $(lsb_release -cs) stable" </del> <ins> $release stable" </ins> # Install Docker apt-get update &&

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : Which jobs are failing: Up W0501 09:18:42.774] scp: /var/log/konnectivity- No such file or directory W0501 09:18:42.775] scp: No such file or directory W0501 09:18:42.775] scp: No such file or directory W0501 09:18:42.775] scp: *: No such file or directory W0501 09:18:42.780] ERROR: () [/usr/bin/scp] exited with return code [1]. /milestone v1.19 /priority critical-urgent /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.19
I am seeing the same error listed above on successful runs as well so I think the issue might be elsewhere
/assign

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : Which jobs are failing: Up W0501 09:18:22.072] scp: /var/log/konnectivity- No such file or directory W0501 09:18:22.072] scp: No such file or directory W0501 09:18:22.072] scp: No such file or directory W0501 09:18:22.073] scp: *: No such file or directory W0501 09:18:22.075] ERROR: () [/usr/bin/scp] exited with return code [1]. /milestone v1.19 /priority critical-urgent /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.19
/assign


Question: #### etcd <del> You need an etcd somewhere in your path. To get from head: </del> <ins> You need an [etcd](https://github.com/coreos/etcd) somewhere in your path. Get the [latest release](https://github.com/coreos/etcd/releases/) and place it in `/usr/bin`. </ins> <del> ``` go get github.com/coreos/etcd go install github.com/coreos/etcd sudo ln -s "$GOPATH/bin/etcd" /usr/bin/etcd # Or just use the packaged one: sudo ln -s "$REPO_ROOT/target/bin/etcd" /usr/bin/etcd ``` </del> ### Starting the cluster

When following the "run locally" guide I ran in to this issue: I found that is the function start_etcd which is calling (something like): I'm on Ubuntu 14.04 and installed etcd using the go get / go install as detailed in . Update: This is because I'm using go get / go install. If you take a look at it says not to use the master branch (breaking changes). You need to install etcd from the releases, not from go get. So I suppose the guide should be updated to point to ?
Hm, I thought we fixed this already? I guess maybe it was wrong in multiple places.
Certainly in the locally guide here: It lists a "go get, go install" way to get etcd
Thanks for the fix!
what exact version of etcd do we need?
If you look in the install- script it shows the revision we chose. Brendan On Oct 25, 2014 9:49 PM, "Ming Fang" wrote:


Question: // TODO(feiskyer): disable-tcp-reset annotations has been depracated since v1.18, it would removed on v1.20. ServiceAnnotationLoadBalancerDisableTCPReset = "service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset" <ins> // ServiceAnnotationLoadBalancerHealthProbeProtocol determines the network protocol that the load balancer health probe use. // If not set, the local service would use the HTTP and the cluster service would use the TCP by default. ServiceAnnotationLoadBalancerHealthProbeProtocol = "service.beta.kubernetes.io/azure-load-balancer-health-probe-protocol" // ServiceAnnotationLoadBalancerHealthProbeRequestPath determines the request path of the load balancer health probe. // This is only useful for the HTTP and HTTPS, and would be ignored when using TCP. If not set, // `/healthz` would be configured by default. ServiceAnnotationLoadBalancerHealthProbeRequestPath = "service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path" </ins> // serviceTagKey is the service key applied for public IP tags. serviceTagKey = "service" // clusterNameKey is the cluster name key applied for public IP tags.

What would you like to be : Support for additional load balancer health probe configurations (specifically HTTP and HTTPS) for Azure Load Balancers. Why is this needed: The generated Azure Load Balancers for type services only support TCP health probes. Those guarantee that the backend pool consists of nodes listening on the target port but do not validate whether or not anything is responding. We've also noticed elevated volumes SSL handshake errors generated by the TCP health checks targeting 443 when using more verbose log level configurations in ingress-nginx. It would be valuable to configure the type of health probe used by the AKS-created Azure Load Balancer.
/sig azure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any update on this?
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about this feature requests?
I had a look at a possible implementation. I did not create a PR yet, because the relevant files are in and the said there should only be bug fixes, no new features. In , there is some logic which would lead to the load balancer created with protocol HTTP (which is what we want) The check is from here: and the "RequestsOnlyLocalTraffic" is about the : However, I do not understand the relationship here, why setting should result in HTTP health check. What I think we could do is add a new annotation like and when this is set to , the health check will use the HTTP protocol in .

We could also add an additional annotation to set the property of the health check probe, this could be used so that people can set the path to or similar.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We have a similar requirement for having an http health probe with a custom path. According to comment, we could get an http probe by setting but our service expects health probes on a different path, not . Looks like proposed solution would work for us too. Are there any workarounds to achieve this currently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign


Question: // Until wraps the watcherClient's watch function with RetryWatcher making sure that watcher gets restarted in case of errors. // The initialResourceVersion will be given to watch method when first called. It shall not be "" or "0" <del> // given the underlying WATCH call issues (#74022). If you want the initial list ("", "0") done for you use ListWatchUntil instead. </del> <ins> // given the underlying WATCH call issues (#74022). </ins> // Remaining behaviour is identical to function UntilWithoutRetry. (See above.) // Until can deal with API timeouts and lost connections. // It guarantees you to see all events and in the order they happened.

has an outdated code comment that refers to the func . That function was removed in Can this comment be updated what should be used instead. TIA /kind documentation
/sig api-machinery
Hi! thank you for your finding! Would you send us a PR and we will review it? it's the most effective way of fixing it! Thank you in advance!
/assign
/assign /unassign Akhilesh2412
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: // ValidateUpdate is the default update validation for an end user. func (priorityLevelConfigurationStrategy) ValidateUpdate(ctx context.Context, obj, old runtime.Object) field.ErrorList { newPL := obj.(*flowcontrol.PriorityLevelConfiguration) <del> oldPL := old.(*flowcontrol.PriorityLevelConfiguration) </del> // 1.28 server is not aware of the roundtrip annotation, and will // default any 0 value persisted (for the NominalConcurrencyShares

In 1.30, Priority and Fairness should allow a zero value, either via v1 or v1beta3 (with the roundtrip annotation) in create and update requests See:
/sig api-machinery /cc /assign
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accpted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: } defer os.Remove(f.Name()) <ins> if _, err := serviceaccount.ReadPrivateKey(f.Name()); err == nil { t.Fatalf("Expected error reading key from empty file, got none") } </ins> if err := ioutil.WriteFile(f.Name(), []byte(rsaPrivateKey), os.FileMode(0600)); err != nil { t.Fatalf("error writing private key to tmpfile: %v", err) }

Kubernetes version Serverversion 1.5.2 (running Hyperkube v1.5.2coreos.1 ()) Environment: Bare-Metal Coreos 1235.6.0 (Kernel 4.7.3-coreos-r2) What happened: kube-controller-manager with ecdsa key fails to start F0127 09:59:04. 1 ] error running controllers: Error reading key for service account token controller: error reading private key file data does not contain a valid RSA or ECDSA private key The key is valid (and works with e.g. the api-server) Command used to create the key:
the issue is the block. removing that (invoke openssl with to omit it from the start) will allow the key to be consumed. will fix in 1.6 to ignore that block when reading the private key block


Question: <ins> apiVersion: v1 kind: ReplicationController metadata: name: first-rc spec: replicas: 1 selector: app: mock template: metadata: labels: app: mock spec: containers: - name: mock-container image: gcr.io/google-containers/pause:2.0 --- apiVersion: v1 kind: ReplicationController metadata: name: second-rc spec: replicas: 1 selector: app: mock template: metadata: labels: app: mock spec: containers: - name: mock-container image: gcr.io/google-containers/pause:2.0 </ins>  No newline at end of file

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign


Question: responseWriteLatencyKey     = "apiserver.latency.k8s.io/response-write" mutatingWebhookLatencyKey   = "apiserver.latency.k8s.io/mutating-webhook" validatingWebhookLatencyKey = "apiserver.latency.k8s.io/validating-webhook" <ins> decodeLatencyKey            = "apiserver.latency.k8s.io/decode-response-object" </ins> ) tracker, ok := LatencyTrackersFrom(ctx)

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } w.probeManager.removeWorker(w.pod.UID, w.container.Name, w.probeType) <del> ProberResults.Delete(w.proberResultsMetricLabels) </del> <ins> ProberResults.Delete(w.proberResultsSuccessfulMetricLabels) ProberResults.Delete(w.proberResultsFailedMetricLabels) ProberResults.Delete(w.proberResultsUnknownMetricLabels) </ins> }() probeLoop:

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.


Question: } opts.Envs = append(opts.Envs, envs...) <ins> // we can only mount individual files (e.g.: /etc/hosts, termination-log files) on Windows only if we're using Containerd. supportsSingleFileMapping := kl.containerRuntime.SupportsSingleFileMapping() </ins> // only podIPs is sent to makeMounts, as podIPs is populated even if dual-stack feature flag is not enabled. <del> mounts, cleanupAction, err := makeMounts(pod, kl.getPodDir(pod.UID), container, hostname, hostDomainName, podIPs, volumes, kl.hostutil, kl.subpather, opts.Envs) </del> <ins> mounts, cleanupAction, err := makeMounts(pod, kl.getPodDir(pod.UID), container, hostname, hostDomainName, podIPs, volumes, kl.hostutil, kl.subpather, opts.Envs, supportsSingleFileMapping) </ins> if err != nil { return nil, cleanupAction, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: When deploying Kubernetes pods, the Linux containers will contain the following entries in the file: # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.240.0.67 dns-querier-1.dns-test-service.e2e-tests-dns- dns-querier-1 However, these are not set on Windows containers; the container's file does not contain any entry. This can be attributed to the fact that individual files cannot be mounted on Windows containers the same way they are on Linux containers. It can be fixed with a change to the kubelet once Windows Server 2019 has shipped, and once CRI-ContainerD is enabled on Windows. We cannot fix it for Windows Server 1803 or earlier with dockershim. What you expected to happen: The file should be managed and populated by Kubelet. How to reproduce it (as minimally and precisely as possible): Spawn a Windows pod, and check its container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cmd /S /C type C:WindowsSystem32driversetchosts Compare the result with a Linux pod's container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cat /etc/hosts Anything else we need to know?: Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: N/A OS (e.g.

from /etc/os-release): Windows Server 1803 (10.0..48) Install tools: acs-engine v0.24.0 Others:
This requires single-file mapping, which won't work in Windows Server version 1803. It should be possible with Windows Server 2019 along with ContainerD
Will it work with Docker on Windows Server 2019?
Not in 18.09 or earlier. I'm not sure what release it will be in because it will require migrating to containerd.
should this issue be tracked for 1.13 in addition to If so please apply the appropriate priority and milestone labels please.
This one needs to be in v1.14. We need to get cri-containerd working first to fix this.
Thanks /milestone v1.14
By default, hosts file are not being used in Windows, it is all registry based. What features is this blocking? CC
The hosts file entries are still taken into account on Windows when it has to resolve a DNS name. This is blocking DNS conformance tests, like.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
Hello! 14 code freeze is coming in about 3 days, is this intended to be implemented in the following week?
No. It is meant to be fixed in the next release, when the Containerd support will be introduced.
/milestone v1.15
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
This is dependent of the ContainerD implementation. If that hasn't been implemented yet, this can't either.
Is there any update is the containerd implementation finished? Are there any relevant issues or PRs that we can use for tracking?
sounds like a no-go for 1.15, let's just move it out as no one is working on it as well. /milestone v1.16
Hello!

I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
/milestone v1.17 We won't be able to do this until containerd is ready. It might work in 1.17
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
We have a PR for it, as you can see above: It;s a pretty small PR too.
The v1.17 code freeze is this Thursday, end of day Pacific time. Will this issue be closed by then?
/milestone v1.18 (code freeze v1.17)
This is going to need a Windows fix, not a ContainerD fix. We won't have that before code freeze on 5 March, so I'm pushing this out to 1.19. /milestone v1.19
This is being tracked with MS internal work item The Windows team thinks an OS fix for Windows Server 2019 will be released with the June patches. I will post updates here as I get them.
Hello from the bug triage team! This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
can you comment on when we can expect a Windows fix for Windows Server 2019 that addresses the single file mount issues in containers when a target file already exists in the container?

we are waiting on a Windows OS fix from Microsoft to proceed here. If that fix comes in time for us to validate this functionality in 1.19 we would like it to be included in 1.19.
Hello! As this issue seems to depend on PR which has been pushed to v1.20, should this issue also be pushed to that milestone?
/milestone 1.20 The OS fix is now expect to be available as part of August Windows patches and then changes to the kubelet will need to follow. /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone.

<detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
yes it is expected be part of August Windows release
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
- Yes, we'd like to make it into 1.20. We were waiting on some bugfixes in the Windows OS and those bugfixes are now available (as of Oct 20 2020).
/remove-lifecycle frozen
/priority important-soon
I believe the work involved here is to update to enable this fucntionality for WIndows if container-runtime == remote (still disable this for DockerShim) Update/Add test coverage Update documentation - possible this section ?
/milestone v1.21 Moving to 1.21 since the required OS fixes were not available in time to implement this functionality in v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello! I am from the Bug Triage team! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
Can this be reopened? From a related problem I have experienced using HostAlias for Windows pod deployments, this issue does not seem to be resolved. Instead, it looks as if it was closed by accident because the conversation went cold.
Hello. What Kubernetes version are you running? Also, on the Windows nodes, do you use docker, or containerd?

This is only supported on Windows nodes with containerd, since it supports single file mapping, which is required in order to mount single files into Windows containers.
We have experimented with both 1.22.11 and a 1.23 release and yea, the container runtime is containerd. Any other ideas to check up on?


Question: return fmt.Errorf("timeout after %v", timeout) } } <ins> func Test_NestedPendingOperations_OperationExists_PendingFirst(t *testing.T) { // Arrange grm := NewNestedPendingOperations(true /* exponentialBackOffOnError */) volumeName := v1.UniqueVolumeName("test-volume") podName1 := volumetypes.UniquePodName("pod1") podName2 := volumetypes.UniquePodName("pod2") podName3 := volumetypes.UniquePodName("pod3") podName4 := EmptyUniquePodName nodeName := EmptyNodeName // delay after an operation is signaled to finish to ensure it actually // finishes before running the next operation. delay := 50 * time.Millisecond // fake operation1 for pod1 failed operation1DoneCh := make(chan interface{}) operation1 := generateWaitWithErrorFunc(operation1DoneCh) err1 := grm.Run(volumeName, podName1, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation1, OperationName: "umount"}) if err1 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err1) } // fake operation2 for pod2 fails operation2DoneCh := make(chan interface{}) operation2 := generateWaitWithErrorFunc(operation2DoneCh) err2 := grm.Run(volumeName, podName2, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation2, OperationName: "umount"}) if err2 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err2) } // fake operation3 for pod3 pending operation3DoneCh := make(chan interface{}) operation3 := generateWaitFunc(operation3DoneCh) defer func() { close(operation3DoneCh) }() err3 := grm.Run(volumeName, podName3, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation3, OperationName: "umount"}) if err3 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err3) } operation1DoneCh <- true operation2DoneCh <- true time.Sleep(delay) // fake operation4 for EmptyUniquePodName should be rejected as operation3 is still pending operation4DoneCh := make(chan interface{}) operation4 := generateWaitFunc(operation4DoneCh) defer func() { close(operation4DoneCh) }() err4 := grm.Run(volumeName, podName4, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation4, OperationName: "mount"}) // Assert if err4 == nil { t.Errorf("NestedPendingOperations did not fail. Expected an operation to already exist") } if !IsAlreadyExists(err4) { t.Errorf("NestedPendingOperations did not return alreadyExistsError, got: %v", err4) } } func Test_NestedPendingOperations_OperationExists_ExactMatchFirstNoPending(t *testing.T) { // Arrange grm := NewNestedPendingOperations(true /* exponentialBackOffOnError */) volumeName := v1.UniqueVolumeName("test-volume") podName1 := volumetypes.UniquePodName("pod1") podName2 := volumetypes.UniquePodName("pod2") podName3 := volumetypes.UniquePodName("pod3") podName4 := EmptyUniquePodName nodeName := EmptyNodeName // delay after an operation is signaled to finish to ensure it actually // finishes before running the next operation. delay := 50 * time.Millisecond backoffDelay := 500 * time.Millisecond // fake operation1 for pod1 fails operation1DoneCh := make(chan interface{}) operation1 := generateWaitWithErrorFunc(operation1DoneCh) err1 := grm.Run(volumeName, podName1, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation1, OperationName: "umount"}) if err1 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err1) } // fake operation2 for pod2 fails operation2DoneCh := make(chan interface{}) operation2 := generateWaitWithErrorFunc(operation2DoneCh) err2 := grm.Run(volumeName, podName2, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation2, OperationName: "umount"}) if err2 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err2) } // fake operation3 for pod3 fails operation3DoneCh := make(chan interface{}) operation3 := generateWaitWithErrorFunc(operation3DoneCh) err3 := grm.Run(volumeName, podName3, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation3, OperationName: "umount"}) if err3 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err3) } operation1DoneCh <- true operation2DoneCh <- true operation3DoneCh <- true time.Sleep(delay) // fake operation4 with EmptyUniquePodName fails operation4DoneCh := make(chan interface{}) operation4 := generateWaitWithErrorFunc(operation4DoneCh) err4 := grm.Run(volumeName, podName4, nodeName /* nodeName */, volumetypes.GeneratedOperations{OperationFunc: operation4, OperationName: "mount"}) if err4 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err4) } operation4DoneCh <- true // operation for pod2 retry time.Sleep(backoffDelay) operation5 := noopFunc err5 := grm.Run(volumeName, podName2, nodeName, volumetypes.GeneratedOperations{OperationFunc: operation5, OperationName: "umount"}) if err5 != nil { t.Errorf("NestedPendingOperations failed. Expected: <no error> Actual: <%v>", err5) } time.Sleep(delay) // Assert // Operation5 will override operation2, since we successfully finished unmount operation on pod2, it should be removed from operations array grm.(*nestedPendingOperations).lock.Lock() defer grm.(*nestedPendingOperations).lock.Unlock() for _, op := range grm.(*nestedPendingOperations).operations { if op.key.podName == podName2 { t.Errorf("NestedPendingOperations failed. Operation for pod2 should be removed") } } } </ins>

on my scenescreate same cronjob use same sfs(the nfs storage on Huawei Cloud) type storage by pvc to do same training work. At the same time, a node will schedule many pods using the same pvc, and there will be many mount and umount tasks about this pvc executed at the same time. And we find all pods on a node at container creating state, because of waiting pvc to mount. check log, we can't find any we can't find any operationExecutor.MountVolume started or failed log like "operationExecutor.MountVolume started for" and also we check volumemanagertotalvolumes metrics, volume is add to dsw. analyze code, if isExpectedError, kubelet not log any thing and operationExecutor will generate operation, add call pendingOperations Run, pendingOperations Run will check operation slice, to find whether has same operation exists, and if operation exist, will check operation's operationPending status, if equal true, means same operation now is excuting, and return AlreadyExists. operationKey composed of volumeName, podName, nodeName so MountVolume operationKey {volumeName, EmptyUniquePodName,EmptyNodeName}, use EmptyUniquePodName to avoid executing mount/map from multiple pods referencing the same volume in parallel soUnmountVolume operationKey {volumeName, podUID,EmptyNodeName}, because All volume plugins can execute unmount/unmap for multiple pods referencing the same volume in parallel now see isOperationExists func, if previousOp.podName equals to EmptyUniquePodName or current operation podName equals to EmptyUniquePodName, will return podNameMatch, mount operation can find umount operation or umount operation can find mount operation because of isOperationExists, this mean if a volume now is excuting mount, a umount can't add to operations, and if a umount now is excuting, a mount operation can't add to operations, but multiple pods referencing the same volume can add multiple umount operation into operation, may lead operation leak.

t0 two pod do umount volume operation t1 pod1 umount failed, update pending to false t2 pod3 add, operation excuter add mount operation, will use index 0, override umount operation t3 mount operation failed , update pending to false t4 pod2 umount operation add, will find index 0, add override opertation, now ,we have two same umount operation, now we have two goroutines to do pod2 umount operations t5 first goroutine pod2 umount failed, will update index 0 operation, update pending to false t6 second goroutine pod2 umount success, will delete index 0 operation, finally lead a umount opertion in cache, and this operation will lead all mount opeartion return AlreadyExistsError, and not cause any csi call. and i think there exists some other scene can trigger this leak. and also, when operation success, deleteOperation, will use tail element to replace, and may increase the probalility of occurrence. because isOperationExists will always pick first match one. and also override mechanism may lead exponentialBackOff invalid for example mount and umount take turns to excute. pv can mount , pod can running reproduce step: create same cronjob that use same pvc(use csi sfs) and csi driver restart periodic No response_ <details</details23 <detailsHuaweiCloud </details<details</details<details</details<details</details<details</detailsCSI
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
+1 meet the same problem
I don't know much about volumeManager. It seems that there is a problem with the override mechanism. Can mount and unmount of the same volume be executed at the same time? If it can, mount should not override any unmount operation. Otherwise, the mount operation should override all matched operations, not just the first one.

also don't know much about design history background, but comments in volume manager seems to conflict with each other among UmountDevice and UnmountVolume. need sameone who familar with it to explain Otherwise, the mount operation should override all matched operations, not just the first one. yes, must discuss rules for the existence of an operation
cc
Suppose there are the following cases: volume's "volumemount" should block the next same "volumemount" or "umountdevice" and all related pods's "volumeumount" exp1: , then could not be , then could be volume's "volumeumount" should block the next same "volumeumount" and it's "volumemount" or "umountdevice", but it should not block it's related other pods's "volumeumount" exp2: , then could not be , then could not be , then could not be , then could be volume's "umountdevice" should block the next same "umountdevice" or "volumemount" and all related pods's "volumeumount" exp3: , then could not be , then could be So we should improve the isOperationExists to satify case 2, that is, the intermediate state should not occur. Maybe since this pr: , this logic remains the same. /cc
/remove-sig node for sig storage to triage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale