Question: if err != nil { t.Fatalf("error adding new service: %#v", err) } <ins> if !svcInfo.isAlive() { t.Fatalf("wrong value for isAlive(): expected true") } </ins> conn, err := net.Dial("tcp", joinHostPort("", svcInfo.proxyPort)) if err != nil { t.Fatalf("error connecting to proxy: %v", err)

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" </ins> "k8s.io/kubernetes/test/e2e/framework" e2ekubelet "k8s.io/kubernetes/test/e2e/framework/kubelet" e2emetrics "k8s.io/kubernetes/test/e2e/framework/metrics"

ci-kubernetes-e2e-capz-master-containerd-windows Kubernetes e2e suite: [It] [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers container stats validation feb 8th 2023 reading of the log is showing reporting zero It looks like the pod had started up just as it check the failure on the test. Adding a retry or a longer wait might make the test more reliable /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: } } EOF <ins> create_ns </ins> $kubectl create -f rm.json --namespace=$NS $kubectl create -f rm-s.json --namespace=$NS sleep 3 # precaution to prevent fe from spinning up too soon.

During the review of , it's better to also update and to use non-default namespace. cc .
cc
/ , I can continue to work on this one :).
/ , can you help to assign this issue to me? So I can trace the tasks in my hand :).
I'd be happy to assign you once you accept the invitation to the org. Do you plan to work on this now? If you're not actively working on it, I'd prefer to leave it unassigned.
, would you help to send the invitation ? Yes, I'd like to work on this now, that's a good start for me to learn example & e2e test suite :).
Already sent. Visit to accept. See
, I'm kuber now , thanks. I'm working on this now :). Checked and , openshift-origin is not using default namespace now; and we can update 's script to update namespace. But it seems and other example are still using default namespace, should we go through all example and update them accordingly?
, should I update all ns in one commit or one commit per example?
In which examples do you plan to make this change? cc
Making this change in some examples would be useful, but only if we explain why we're doing it. Additionally, linking to existing docs and blogs about namespaces would be helpful: I don't want to do this in all of the examples, because it does add an extra step, and labels also provide a means of managing a group of related resources within a namespace. Perhaps could recommend examples where this would make sense.


Question: bindableNodeLabels := cliflag.ConfigurationMap(c.NodeLabels) fs.Var(&bindableNodeLabels, "node-labels", "Additional node labels") fs.Var(utiltaints.NewTaintsVar(&c.RegisterWithTaints), "register-with-taints", "Register the node with the given list of taints (comma separated "<key>=<value>:<effect>"). No-op if register-node is false.") <ins> fs.IntVar(&c.MaxPods, "max-pods", maxPods, "Number of pods that can run on this Kubelet.") bindableExtendedResources := cliflag.ConfigurationMap(c.ExtendedResources) fs.Var(&bindableExtendedResources, "extended-resources", "Register the node with extended resources (comma separated "<name>=<quantity>")") </ins> } func (c *hollowNodeConfig) createClientConfigFromFile() (*restclient.Config, error) {

A kubernetes cluster is normally constructed from a variety of machine templates. Different instance groups may have different CPU, memory capacity. They may be equipped with specialized accelerators, network devices. I think we should have an extensible way for users to custom node configuration to simulate real production cluster states. Currently, nodelabel and nodetaints are supported. I am consider to accept a template with more configuration like resources so we won't grow option list in the future. Without this feature, it's hard for users to simulate heterogeneous cluster.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scalability


Question: }, expected: true, }, <ins> { name: "with changes on Conditions", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-0", }, Status: v1.PodStatus{ Conditions: []v1.PodCondition{ {Type: "foo"}, }, }, }, isAssumedPodFunc: func(*v1.Pod) bool { return true }, getPodFunc: func(*v1.Pod) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-0", }, } }, expected: true, }, </ins> } { t.Run(test.name, func(t *testing.T) { c := &Scheduler{

What happened: An extra scheduling attempt is observed in some scenarios when kube-scheduler already has already assumed a pod and this has entered the binding routine. If, during this time, a pod update is received, the pod is back to the scheduling queue, even if binding succeeds. This is the correct behavior, as scheduling conditions might change. However this updates could just be unrelated status updates, such as updates to pod conditions. We observe an extra scheduling attempt, which is visible in the log line: Which indicates that an attempt was made when the pod was already scheduled. What you expected to happen: No extra scheduling attempts How to reproduce it (as minimally and precisely as possible): This depends on other controllers running on the cluster that might report pod conditions. One such controller is
/sig scheduling


Question: } } func Test_v1_PodStatus_to_core_PodStatus(t *testing.T) { <del> // fail v1FailTestInputs := []v1.PodStatus{ </del> <ins> asymmetricInputs := []struct { name string in   v1.PodStatus out  core.PodStatus }{ </ins> { <del> PodIP: "1.1.2.1", // fail becaue PodIP != PodIPs[0] PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, </del> <ins> name: "mismatched podIP", in: v1.PodStatus{ PodIP: "1.1.2.1", // Older field takes precedence for compatibility with patch by older clients PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, out: core.PodStatus{ PodIPs: []core.PodIP{ {IP: "1.1.2.1"}, }, }, }, { name: "matching podIP", in: v1.PodStatus{ PodIP: "1.1.1.1", PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, out: core.PodStatus{ PodIPs: []core.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, }, { name: "empty podIP", in: v1.PodStatus{ PodIP: "", PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, out: core.PodStatus{ PodIPs: []core.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, </ins> }, }, } // success v1TestInputs := []v1.PodStatus{ // only Primary IP Provided

What happened: The cluster master in a Kubernetes v1.14.10 cluster was upgraded to V1.16.7. The process of upgrading nodes is under way and has been spread over a week so far. The cluster owner reported seeing a high percentage of pod status PATCH failures. The apiserver logs show errors like: The errors are all from the v1.14 nodes and have all been for pods associated with DaemonSets. We don't yet know why pods' IPs are changing, but it appears that pod status patches that include a changed result in an object where contains the old IP, and has the new IP. We believe this is because was introduced in Kubernetes v1.15 and the patch request from a v1.14 node results in both and being present in the result. If you aren't monitoring for request failures it may not obvious that any error is occurring: The pods continue to show their last successfully reported state. Deleting such pods, however, results in the pod being stuck in a state. We are only seeing this for DaemonSet pods - I suspect because whatever resulted in pod IPs changing also causes non-DaemonSet pods to be deleted and recreated. We have a number of cluster still on v1.14 that we expect to upgrade to v1.15 or v1.16. What you expected to happen: Kubernetes v1.14 nodes should be able to update pod status during the transition from v1.14 to v1.16 even if pod IPs change during that time. How to reproduce it (as minimally and precisely as possible): We have recreated this in the IBM Cloud Kubernetes Service by "reloading" nodes. The reload process leaves the Kubernetes node object in place, and essentially reloads the image with the proper kubeconfig to rejoin the cluster. The cluster I was debugging does not appear to have had nodes "reloaded" - something else is triggering the pod IP changes. Create a v1.14 cluster Deploy a DaemonSet (anything with pod networking should suffice) Upgrade cluster master to v1.16 Reload a v1.14 node. One the node is back up, shows the pod's network IP has changed. Kubelet and apiserver logs show the errors describe earlier. Anything else we need to know?

: Environment: Kubernetes version (use ): v1.14.x (nodes) and v1.16.7 (master) Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): 18.04.4 LTS Kernel (e.g. ): 4.15.0-76-generic Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/sig network
cc
/priority critical-urgent if it's possible for podIPs to change and the server defaults to , then the API server has to treat the singular pod IP as authoritative if there's a mismatch or the patch case breaks
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-sig api-machinery


Question: } func (h *EtcdHelper) listEtcdNode(key string) ([]*etcd.Node, uint64, error) { <del> result, err := h.Client.Get(key, false, true) </del> <ins> result, err := h.Client.Get(key, true, true) </ins> if err != nil { index, ok := etcdErrorIndex(err) if !ok {

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: } fw.updateReader = startReading(fw.updateChan) fw.fakeKubelet = &fakeKubelet{} <del> fw.serverUnderTest = &Server{ host:    fw.fakeKubelet, updates: fw.updateChan, } </del> <ins> server := NewServer(fw.fakeKubelet, fw.updateChan) fw.serverUnderTest = &server </ins> fw.testHTTPServer = httptest.NewServer(fw.serverUnderTest) return fw }

The Kubelet REST end-point has turned into a complicated switch-case code block that is harder to maintain overtime. The Kubelet should move to a http.ServeMux pattern and register handler functions to a pattern.
I am working on a PR to submit in next couple of days that will make this change, so feel free to assign to me if possible.
Tagging because I asked for this in one of his PRs
This one is also under my radar. If no one had time, feel free to assign it to me too.
I have a branch in progress. Feel free to offer feedback on it when I submit. Sent from my iPhone


Question: // +genclient:nonNamespaced // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // +k8s:prerelease-lifecycle-gen:introduced=1.7 <del> // +k8s:prerelease-lifecycle-gen:deprecated=1.19 </del> <ins> // +k8s:prerelease-lifecycle-gen:deprecated=1.16 // +k8s:prerelease-lifecycle-gen:removed=1.22 </ins> // +k8s:prerelease-lifecycle-gen:replacement=apiextensions.k8s.io,v1,CustomResourceDefinition // CustomResourceDefinition represents a resource that should be exposed on the API server.  Its name MUST be in the format // <.spec.name>.<.spec.group>. <del> // Deprecated in v1.16, planned for removal in v1.19. Use apiextensions.k8s.io/v1 CustomResourceDefinition instead. </del> <ins> // Deprecated in v1.16, planned for removal in v1.22. Use apiextensions.k8s.io/v1 CustomResourceDefinition instead. </ins> type CustomResourceDefinition struct { metav1.TypeMeta   `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`

The following code snippet is very confusing: Is v1beta1 deprecated in 1.16 or 1.19? Is it to be removed in 1.19 or 1.22? Please clarify the exact versions so the many users of CRD v1beta1 APIs know what to do.
/sig api-machinery
/kind deprecation
/unkind feature
Deprecated in 1.16. Planned to be removed in 1.22. We'll update the docs to be consistent before 1.19 is released.
/assign


Question: reasons = append(reasons, &PredicateFailureError{r.Name, r.Reason}) } } <ins> // Check taint/toleration except for static pods if !types.IsStaticPod(pod) { _, isUntolerated := corev1.FindMatchingUntoleratedTaint(nodeInfo.Node().Spec.Taints, pod.Spec.Tolerations, func(t *v1.Taint) bool { // Kubelet is only interested in the NoExecute taint. return t.Effect == v1.TaintEffectNoExecute }) if isUntolerated { reasons = append(reasons, &PredicateFailureError{tainttoleration.Name, tainttoleration.ErrReasonNotMatch}) } } </ins> return reasons }

The method doesn't check taints and tolerations. This is problematic in 2 scenarios: If the users bypass scheduler by setting a value in If the node restarts. The node would try to restart scheduled pods once it comes back up, even if it's not ready (has a not-ready taint). Kubelet to check taints and tolerations. a taint in a node. a Pod with set to the Node above, without a matching toleration. /sig node /sig scheduling
this reminds me of your kubelet sync PR ...
I don't think this is a syncing issue. The logic doesn't check taints at all.
scenario 1 is pretty bad, i will let the #sig-node slack know about this.
It doesn't sound like a syncing issue, but we seem to have multiple issues with pod admission/management on restart currently
I'm aware of that issue. This should be unrelated.
I would like to address this. /assign
I think it would be great to proceed after merging ref:
Please add me as reviewer
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale PR in review:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: } wg.Add(1) go func(extIndex int) { <del> defer wg.Done() </del> <ins> metrics.SchedulerGoroutines.WithLabelValues("prioritizing_extender").Inc() defer func() { metrics.SchedulerGoroutines.WithLabelValues("prioritizing_extender").Dec() wg.Done() }() </ins> prioritizedList, weight, err := extenders[extIndex].Prioritize(pod, nodes) if err != nil { // Prioritization errors from extender can be ignored, let k8s/other extenders determine the priorities

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track number of scheduler binding goroutines. This will be a generic metric with as the only label for now. metric name Description -- -- scheduler_goroutines Number of running binding goroutines Why is this needed: Helps us track how many binding are running and monitor issues such as goroutine leak. /sig scheduling
/assign


Question: return match } <ins> func hasSrcType(rules []iptablestest.Rule, srcType string) bool { for _, r := range rules { if r[iptablestest.SrcType] != srcType { continue } return true } return false } </ins> func TestHasJump(t *testing.T) { testCases := map[string]struct { rules     []iptablestest.Rule

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: This is discussed on When externalTrafficPolicy: Local is used, in GKE, external traffic coming in to the Load Balancer works correctly. But traffic that originates within the GKE cluster is broken, a TCP SYN is not responded to. My use case: I run gitlab, gitlab-runner, docker-registry, wiki inside kubernetes. But I also access the docker-registry outside kubernetes. So my hostname for it is ''. In DNS this resolves to the LB IP. Actually, all 3 services have DNS names that resolve to the same LB IP, which then goes to the nginx ingress. so , , are generally addressed as LB-IP w/ HOSTNAME+SNI. Its 'astonishing' to not be able to access this DNS name when I'm 'inside' but can when I'm 'outside'. If I enable the IP transparency (because in my wiki I want the IP of the user who made the edit), then my gitlab-runner fails(since it cannot pull from the registry). Tim Hockin has this to say: and then: What you expected to happen: My expectation is that it should not matter where my traffic originates, if it is to a globally-routeable public IP it should work. How to reproduce it (as minimally and precisely as possible): Configure a Load Balancer, an ingress, and 2 services. Have service 2 talk to service one using a DNS name that maps to the Load Balancer. Anything else we need to know?

: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-05-10T17:23:18Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Google GKE OS (e.g. from /etc/os-release): BUILDID=.89.0 NAME="Container-Optimized OS" KERNELCOMMITID= GOOGLECRASHID=Lakitu VERSIONID=66 PRETTYNAME="Container-Optimized OS from Google" VERSION=66 GOOGLEMETRICSPRODUCTID=26 HOMEURL="" ID=cos Kernel (e.g. ): Linux gke-k8s-default-pool--d3ft 4.14.22+ SMP Thu May 10 17:54:42 PDT 2018 x8664 Intel(R) Xeon(R) CPU @ 2.00GHz GenuineIntel GNU/Linux Install tools: Others:
/sig network
Agree that this is weird and feels left-out. The fix is not complicated but I need an owner to carry it through testing.
if nobody is working on it can I take it as my first issue?.
Is anybody working on this issue. Seems like ipvs might have the same problem.
/assign
I believe I'm running into this problem and I think adding as Tim mentioned would fix it. are you working on this? If not I don't mind taking a stab at it :)
Thanks, Sorry I kinda forgot this thing. I was fixed this thing just like you said But encounter some problem when testing in gce environment, which caused by route rules. Sorry I forgot the detail, can you do some test in your environment?
It seems the pull request is stalled on this. Does anyone have any input?
Being relied on by istio
Following this.
Did this fix get lost in the shuffle?

Any chance getting this merged in? This also impacts Digital Ocean's Kubernetes + LB setups out of the box.
As far as I know it hasn't
Merge On Mon, Feb 18, 2019, 7:58 PM Jarred Capellman wrote:
still affecting me on gke. i want to enable the so that I can use the spam-blocking in wordpress (which is by ext ip), but I also want to refer to other services which may be internal to the cluster via their proper name (to get the `tls etc).
Same here.
I have more or less spelled out the fix, but I don't have teh bandwidth to polish it off right now. I am looking for volunteers...
I can take this given I've run into this a while back /assign
I was only able to reproduce this when trying to reach the public IP from the host network. Reaching the public IP from inside the pod network is working (as per this ). It's possible I missed a step in trying to reproduce this but wanted to confirm with you first. Are you seeing this for traffic originating only from host network or does it also apply for pods in the cluster network?
Hi. My situation. I have a DNS name (git.MYDOMAIN). It runs on a self-hosted gitlab via nginx-ingress via LoadBalancer in GKE. Other pods of mine sometimes reference https://git.MYDOMAIN. They may happen to run in the same cluster as git.MYDOMAIN, they may run in a different cluster. When they run in the same cluster on GKE, they fail if is enabled. Specifically, since the DNS resolves to (My external Loadbalancer IP) an IP outside of GKE, it goes out, hits the LB, and then somewhere along the reverse path is lost. I have calico enabled (as per GKE managed). I am not using Istio. I am using a cluster that predates the new private IP ones. I have not tried disabling the LoadBalancer and just using node ip for the ingress. I have not tried to see if there is any difference if the Pod trying to reach my git is on the same node as it or not.

Based on the proxy rules (may be different at this point cause I'm looking at master), I don't think your traffic is even reaching the LB, it's just being dropped by the iptable rules on the host. My original question was, are the pods trying to reach the same-cluster external IP have , if yes, can you reproduce on pods not on host network?
I do not have set.
Are you still on v1.10?
this is the most recent version on gke afaik.
I opened a fix here based on the comments and some testing I did on my GKE cluster. It needs more validation though
Again, I wasn't able to reproduce this for pods that are within the cluster CIDR (but I could on the host network). The rule that allows for pods inside the cluster CIDR to still reach pods behind an LB is this: Are you able to check if a rule like ^ exists on your cluster?
Here's an example, right now (www is hosted on ingress w/ loadbalancer w/ local policy). Note: this example is from the node (not a pod).
Here's the full rules:
^ can you show me the same test from within a pod (not host network)? I have a fix for within a node but I'd like to make sure it's reproducible inside a pod before I dig deeper
Interestingly its working in a Pod on this cluster now. (?!) I guess some previous upgrade of k8s on gke has altered it? on cannot reach on on can reach. hmmm.
ok w/o the node being able to reach this i'm still blocked (since my container registry might run there, and thus kubelet cannot reach, so my gitlab CI pipeline cannot reach). but, i guess, now the pods w/o hostnetwork can, so some progress
Thanks for checking should fix the internal reachability issue from the host. Though it requires SNAT.
Hello Maybe better way is to check if and then skip to KUBE-SERVICES chain? So, request will be sent to loadbalancer and source IP will be preserved. Can you try this?
Solution described above will affect traffic from pods, it will not be . Then maybe we can replace with something to send traffic to loadbalancer?

I think this will break existing implementations, the fwchain is what routes the LB IPs to the correct service chain. Not all LB implementations (think MetalLB) depend on node port for routing
I tried to exclude node IP with from on node without local endpoints and it works! What do you think about this solution? I tested with:
I believe there's a similar rule for that already but it doesn't check source IP If you jump to FW chain, you end up in XLB chain which drops traffic if there's no local endpoints. Does your test also work on nodes where there's no local endpoints? Can you also verify that your traffic actually goes to the LB and is not being masqueraded through the service chain?
Yes, I just source IP check to this rule on node without local endpoints, so local traffic doesn't jump to FW, it doesn't drops in XLB and it goes to LB. Of course, if you do this on node with local endpoints you will not reach that endpoint (at least when using Metallb). This solution doesn't work if nodes haven't network access to LB. Your solution haven't this problem. For me it is more important then saving source IP, so now I think your solution is better. But why you add jump to MARK-MASQ before jump to SVC when each SEP chain already contains jump to MARK-MASQ? I tested your solution with:
Good point, I think might be redundant, will update the PR soon :)


Question: ## Changelog since v1.14.4 <del> **No notable changes for this release** </del> <ins> * Fix CVE-2019-11249: Incomplete fixes for CVE-2019-1002101 and CVE-2019-11246, kubectl cp potential directory traversal ([#80436](https://github.com/kubernetes/kubernetes/pull/80436)) * Fix CVE-2019-11247: API server allows access to custom resources via wrong scope ([#80750](https://github.com/kubernetes/kubernetes/pull/80750)) See also the [security announcement for this release](https://groups.google.com/forum/#!topic/kubernetes-security-announce/vUtEcSEY6SM). </ins> # v1.14.4

What would you like to be : The release notes state "No notable changes for this release": However it turns out these releases fix important security problems: Why is this needed: If people track releases in, for example, GitHub and then use the release notes to see what's changed, they will totally miss that important security problems have been fixed in these releases. People should not have to look in multiple places to find news of such fixes. The current practice risks people not upgrading because there were "no notable changes" and being oblivious to the risks they run.
I expect this is one for and probably others but I don't really have any clue who.
/sig release This has to do with the timing of cutting the releases vs sending the announcements. The notices were included in the announcement, but not in the generated md files. /assign can you open a PR to update those?
Looks like they put it in the email and the group, but not the release notes. !topic/kubernetes-security-announce/vUtEcSEY6SM/discussion
Just opened a PR to fix this: (I only noticed that this issue was already assigned after committing my changes, hope that's OK)
Thanks that PR looks exactly like what I'm after for this particular update!


Question: "github.com/onsi/ginkgo" ) <del> var _ = SIGDescribe("[Feature:Windows] Cpu Resources", func() { </del> <ins> var _ = SIGDescribe("[Feature:Windows] Cpu Resources [Serial]", func() { </ins> f := framework.NewDefaultFramework("cpu-resources-test-windows") // The Windows 'BusyBox' image is PowerShell plus a collection of scripts and utilities to mimic common busybox commands

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig windows /cc /priority critical-urgent /milestone v1.18
/assign
I think we might need to run this job in Serial. This test is flakey and fails the same way each time. Looking at the logs for the failures I see we try to start a single pod ï¿½[1mSTEPï¿½[0m: Creating one pod with limit set to '0.5' ï¿½[1mSTEPï¿½[0m: Creating one pod with limit set to '500m' But then events captured in the logs show the pod getting rescheduled to a different node ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "cpu-resources-test-windows-3448". ï¿½[1mSTEPï¿½[0m: Found 8 events. Mar 17 14:08:03.918: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for cpulimittest--30aa-4ac7-9ad2-: {default-scheduler } Scheduled: Successfully assigned cpu-resources-test-windows-3448/cpulimittest--30aa-4ac7-9ad2- to 2861k8s001 Mar 17 14:08:03.918: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for cpulimittest--2fea-4e58-8dc7-: {default-scheduler } Scheduled: Successfully assigned cpu-resources-test-windows-3448/cpulimittest--2fea-4e58-8dc7- to 2861k8s000 Mar 17 14:08:03.918: INFO: At 2020-03-17 14:05:44 +0000 UTC - event for cpulimittest--30aa-4ac7-9ad2-: {kubelet 2861k8s001} Pulled: Container image "e2eteam/busybox:1.29" already present on machine Mar 17 14:08:03.

918: INFO: At 2020-03-17 14:05:44 +0000 UTC - event for cpulimittest--30aa-4ac7-9ad2-: {kubelet 2861k8s001} Created: Created container cpulimittest--30aa-4ac7-9ad2- Mar 17 14:08:03.918: INFO: At 2020-03-17 14:05:46 +0000 UTC - event for cpulimittest--30aa-4ac7-9ad2-: {kubelet 2861k8s001} Started: Started container cpulimittest--30aa-4ac7-9ad2- Mar 17 14:08:03.918: INFO: At 2020-03-17 14:05:56 +0000 UTC - event for cpulimittest--2fea-4e58-8dc7-: {kubelet 2861k8s000} Pulled: Container image "e2eteam/busybox:1.29" already present on machine Mar 17 14:08:03.918: INFO: At 2020-03-17 14:05:56 +0000 UTC - event for cpulimittest--2fea-4e58-8dc7-: {kubelet 2861k8s000} Created: Created container cpulimittest--2fea-4e58-8dc7- Mar 17 14:08:03.918: INFO: At 2020-03-17 14:05:59 +0000 UTC - event for cpulimittest--2fea-4e58-8dc7-: {kubelet 2861k8s000} Started: Started container cpulimittest--2fea-4e58-8dc7- Mar 17 14:08:03.977: INFO: POD NODE PHASE GRACE CONDITIONS Mar 17 14:08:03.

977: INFO: cpulimittest--30aa-4ac7-9ad2- 2861k8s001 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:05:38 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:05:48 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:05:48 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:05:38 +0000 UTC }] Mar 17 14:08:03.977: INFO: cpulimittest--2fea-4e58-8dc7- 2861k8s000 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:05:49 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:06:02 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:06:02 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-03-17 14:05:49 +0000 UTC }] Mar 17 14:08:03.977: INFO: Mar 17 14:08:04.093: INFO:
/cc
Yes. That's most probably the case. I am preparing a PR for this.
/reopen The job is still red. It seems that was merged in the wrong branch. Can you check? This job runs in branch as well? Do we need to cherry-pick it?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Any news?
is the cherry-pick PR to bring the fix/change into 1.18 branches
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. .PHONY: cauldron TAG = 1.0 cauldron: GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -a -installsuffix cgo --ldflags '-w' cauldron.go container:	cauldron docker build --pull -t kubernetes/cauldron:$(TAG) . push: docker push kubernetes/cauldron:$(TAG) rc: kubectl create --validate -f cauldron-rc.yaml stop: kubectl delete rc cauldron get: kubectl get rc,pods -l app=cauldron scale20: kubectl scale rc cauldron --replicas=20 scale1: kubectl scale rc cauldron --replicas=1 clean: rm -rf cauldron </del>

We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) Apparently the repo is referenced for docker push and pull in It's unclear to me whether we have broken anything, or if this is dead code. I would use to find out if this is referenced anywhere (aside from sundry vendor dirs) If dead code: remove it If used: move to test/e2e/images and follow the same pattern there used to push to /sig testing /area test /wg k8s-infra /priority important-soon /help /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) /area test /wg k8s-infra /priority important-soon /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func TestComposeDNSSearch(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) kubelet := testKubelet.kubelet recorder := record.NewFakeRecorder(20) kubelet.recorder = recorder pod := podWithUidNameNs("", "test_pod", "testNS") kubelet.clusterDomain = "TEST" testCases := []struct { dnsNames     []string hostNames    []string resultSearch []string events       []string }{ { []string{"testNS.svc.TEST", "svc.TEST", "TEST"}, []string{}, []string{"testNS.svc.TEST", "svc.TEST", "TEST"}, []string{}, }, { []string{"testNS.svc.TEST", "svc.TEST", "TEST"}, []string{"AAA", "svc.TEST", "BBB", "TEST"}, []string{"testNS.svc.TEST", "svc.TEST", "TEST", "AAA", "BBB"}, []string{ "Found and omitted duplicated dns domain in host search line: 'svc.TEST' during merging with cluster dns domains", "Found and omitted duplicated dns domain in host search line: 'TEST' during merging with cluster dns domains", }, }, { []string{"testNS.svc.TEST", "svc.TEST", "TEST"}, []string{"AAA", strings.Repeat("B", 256), "BBB"}, []string{"testNS.svc.TEST", "svc.TEST", "TEST", "AAA"}, []string{"Search Line limits were exceeded, some dns names have been omitted, the applied search line is: testNS.svc.TEST svc.TEST TEST AAA"}, }, { []string{"testNS.svc.TEST", "svc.TEST", "TEST"}, []string{"AAA", "TEST", "BBB", "TEST", "CCC", "DDD"}, []string{"testNS.svc.TEST", "svc.TEST", "TEST", "AAA", "BBB", "CCC"}, []string{ "Found and omitted duplicated dns domain in host search line: 'TEST' during merging with cluster dns domains", "Found and omitted duplicated dns domain in host search line: 'TEST' during merging with cluster dns domains", "Search Line limits were exceeded, some dns names have been omitted, the applied search line is: testNS.svc.TEST svc.TEST TEST AAA BBB CCC", }, }, } fetchEvent := func(recorder *record.FakeRecorder) string { select { case event := <-recorder.Events: return event default: return "No more events!" } } for i, tc := range testCases { dnsSearch := kubelet.formDNSSearch(tc.hostNames, pod) if !reflect.DeepEqual(dnsSearch, tc.resultSearch) { t.Errorf("[%d] expected search line %#v, got %#v", i, tc.resultSearch, dnsSearch) } for _, expectedEvent := range tc.events { expected := fmt.Sprintf("%s %s %s", v1.EventTypeWarning, "DNSSearchForming", expectedEvent) event := fetchEvent(recorder) if event != expected { t.Errorf("[%d] expected event '%s', got '%s", i, expected, event) } } } } </ins> func TestCleanupBandwidthLimits(t *testing.T) { testPod := func(name, ingress string) *v1.Pod { pod := podWithUidNameNs("", name, "")

Kubernetes version v1.3.0 Kubelet concatenates host and cluster domain entries for my pod, resulting in an invalid config (too many entries with duplicates). The max is only 6 search domains. Any more and DNS resolution breaks when trying to leverage search domains. The example below has only cluster DNS entries, but illustrates the issue: Host search nameserver 10.233.0.2 nameserver 10.90.0.1 options timeout:2 options attempts:2 Container kubectl run -i --tty busybox --image=busybox -- sh (skip waiting text) / # cat search nameserver 10.233.0.2 options attempts:2 options ndots:5 I should be enable cluster DNS config in my host DNS settings and still have functional pod DNS. Kubelet should be intelligent enough to filter out duplicate search domain entries (or drop search domains beyond the 6th entry)
I think some sane checks needs to be too. libc limits search field to 6 records and 255(?) characters. This limitation should be checked before creation.
Hey guys, I'm working on it.
Dup of


Question:
targetPort

Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod’s container ports. If this is not specified, the value of Port is used (an identity map). Defaults to the service port. More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#defining-a-service

Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod’s container ports. If this is not specified, the value of the port field is used (an identity map). This field is ignored for services with clusterIP=None, and should be omitted or set equal to the port field. More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#defining-a-service

false

string

In 1.0 and 1.1, headless services that specified a mismatched port/targetPort were tolerated (the targetPort was ignored). tightened validation to reject these services. That causes problems when upgrading from currently-working clusters.
2 candidate to avoid breaking backwards compatibility
oh hell. Yeah. I don't think this is worth a breaking change. We should either roll that back ASAP or change it to nuke any user-provided value and replace it with the value. I think the latter is potentially dangerous and also not worth the risk, so I think we should rollback and just update the docs for targetPort. objections?
Who has time to do this? It doesn't look like an auto-rollback is doable.
Already opened a PR in
Yes, I agree this isn't worth a breaking change, But could we give an warning log that targetPort isn't equal to port and would be ignored?
There's no facility in validation for warning. I don't think spamming the server log in response to API requests is that helpful here
Yes, that sounds reasonable. I just think that targetPort was ignored is a little unfriendly to user.


Question: return ${success} } <del> if [ ! $(nc -vz "${host} ${port}") ]; then </del> <ins> if [ ! $(nc -vz "${host}" "${port}") ]; then </ins> kube::log::status "${host}:${port} is free, proceeding..." return ${success} else

It seems like an issue similar to the problem with API server. I'll extend the timeout in the similar fashion. cc
Extending timeout didn't help. It's probably something else.
cc
I think the logs make it pretty clear what went wrong: or does one of you have time to fix this?
The Jenkins docker test image probably needs netcat to be installed. Should be an easy fix; I can take this.
Thanks!
Ugh... neverending story.
This seems like a different problem.
Oh, I see, this actually is the same, but harder: I think the most obvious thing to do is to let kubectl select the port and report back?
reports more occurances. Netcat somehow is no longer installed?
those two are older builds, using , which didn't have netcat yet.
ah thanks!
are you still working on this? Would you like me to take it?
I'm not right now, sorry, feel free to take it.


Question: driver2 := &storagev1.CSIDriver{ ObjectMeta: metav1.ObjectMeta{ Name: "inline-driver-" + string(uuid.NewUUID()), <ins> Labels: map[string]string{ "test": f.UniqueName, }, </ins> }, Spec: storagev1.CSIDriverSpec{

https://k8s- CSIInlineVolumes should support ephemeral VolumeLifecycleMode in CSIDriver API 8/5 https://k8s- No response /sig storage
/triage accepted
/assign


Question: func parseEnvs(envArray []string) ([]api.EnvVar, error) { envs := []api.EnvVar{} for _, env := range envArray { <del> parts := strings.Split(env, "=") if len(parts) != 2 || !validation.IsCIdentifier(parts[0]) || len(parts[1]) == 0 { </del> <ins> pos := strings.Index(env, "=") if pos == -1 { </ins> return nil, fmt.Errorf("invalid env: %v", env) } <del> envVar := api.EnvVar{Name: parts[0], Value: parts[1]} </del> <ins> name := env[:pos] value := env[pos+1:] if len(name) == 0 || !validation.IsCIdentifier(name) || len(value) == 0 { return nil, fmt.Errorf("invalid env: %v", env) } envVar := api.EnvVar{Name: name, Value: value} </ins> envs = append(envs, envVar) } return envs, nil

I'm running application with several env variables. One of them has a key ending with ==. Another variable has a comma separated list of email addresses. While perfectly valid on *nix and windows, we can't pass these values into Kubernetes run command. It fails with "error: invalid env:" Example command: The parsing in should be more robust
I volunteer myself to fix this.
Maybe they can be passed from file? I mean s'rsly.. you need to lits 20 variables by hand?? I've specified them in yaml file already, but kubectl-run have no way to specify it.. without this, deployment is totally impossible when you need to run migrations first!


Question: case "GetInodes": _, err = GetInodes(testcase.path) default: <del> t.Errorf("Case %v (%s, %v) unknown operation %s", seq, testcase.path, true, testcase.op) </del> <ins> t.Errorf("Case %v (%s, %s, %v) unknown operation %s", seq, testcase.name, testcase.path, true, testcase.op) </ins> return true } if err != nil && testcase.expectsSetQuota { fail = true <del> t.Errorf("Case %v (%s, %v) %s expected to clear quota but failed %v", seq, testcase.path, true, testcase.op, err) </del> <ins> t.Errorf("Case %v (%s, %s, %v) %s expected to clear quota but failed %v", seq, testcase.name, testcase.path, true, testcase.op, err) </ins> } else if err == nil && !testcase.expectsSetQuota { fail = true <del> t.Errorf("Case %v (%s, %v) %s expected not to clear quota but succeeded", seq, testcase.path, true, testcase.op) </del> <ins> t.Errorf("Case %v (%s, %s, %v) %s expected not to clear quota but succeeded", seq, testcase.name, testcase.path, true, testcase.op) </ins> } return fail }

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: Help:      "Latency in microseconds for relisting pods in PLEG.", }, ) <ins> PLEGDiscardEvents = prometheus.NewCounterVec( prometheus.CounterOpts{ Subsystem: KubeletSubsystem, Name:      PLEGDiscardEventsKey, Help:      "The number of discard events in PLEG.", }, []string{}, ) </ins> PLEGRelistInterval = prometheus.NewSummary( prometheus.SummaryOpts{ Subsystem: KubeletSubsystem,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it.

The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too.

We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.


Question: } } <ins> // Test_RemoveNodeFromAttachUpdates_Positive expects an entire node entry to be removed // from nodesToUpdateStatusFor func Test_RemoveNodeFromAttachUpdates_Positive(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") nodeToUpdate := nodeToUpdateStatusFor{ nodeName:                  nodeName, statusUpdateNeeded:        true, volumesToReportAsAttached: make(map[v1.UniqueVolumeName]v1.UniqueVolumeName), } asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate // Act err := asw.RemoveNodeFromAttachUpdates(nodeName) // Assert if err != nil { t.Fatalf("RemoveNodeFromAttachUpdates should not return error, but got: %v", err) } if len(asw.nodesToUpdateStatusFor) > 0 { t.Fatal("nodesToUpdateStatusFor should be empty as its only entry has been deleted.") } } // Test_RemoveNodeFromAttachUpdates_Negative_NodeDoesNotExist expects an error to be thrown // when nodeName is not in nodesToUpdateStatusFor. func Test_RemoveNodeFromAttachUpdates_Negative_NodeDoesNotExist(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") nodeToUpdate := nodeToUpdateStatusFor{ nodeName:                  nodeName, statusUpdateNeeded:        true, volumesToReportAsAttached: make(map[v1.UniqueVolumeName]v1.UniqueVolumeName), } asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate // Act err := asw.RemoveNodeFromAttachUpdates("node-2") // Assert if err == nil { t.Fatal("RemoveNodeFromAttachUpdates should return an error as the nodeName doesn't exist.") } if len(asw.nodesToUpdateStatusFor) != 1 { t.Fatal("The length of nodesToUpdateStatusFor should not change because no operation was performed.") } } // Test_RemoveNodeFromAttachUpdates_Negative_Empty expects an error to be thrown // when a nodesToUpdateStatusFor is empty. func Test_RemoveNodeFromAttachUpdates_Negative_Empty(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } // Act err := asw.RemoveNodeFromAttachUpdates("node-1") // Assert if err == nil { t.Fatal("RemoveNodeFromAttachUpdates should return an error as nodeToUpdateStatusFor is empty.") } if len(asw.nodesToUpdateStatusFor) != 0 { t.Fatal("The length of nodesToUpdateStatusFor should be 0.") } } </ins> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,

kubernetes version: 1.5.3 platform: aws deploy tool: kops my master's /var/log/kube-controller- is receiving this type of log: lots of times per second, this node "ip-172-20-114-85.ec2.internal" does not exist in the cluster anymore and is not shown if I shouldn't it be removed from status_updater?
Any work around? Any way to purge those old nodes from the NodeInformer cache?
AFAIK, sig-node doesn't maintain the node controller. Based on the files, the only approver is
It has nothing to do with the NodeController: cc
This was happening in our clusters as well, only the leader was doing it and and killing the container so it restarted made it stop happening.
did you kill the container, which then gets restarted by the scheduler, and that fixed it?
correct
Works for me
Same controller-manager log here using k8s 1.5. I deleted one of the nodes which ran out of resources and became stuck, something I'm still trying to pinpoint the cause of but it appears to be related to volumes.
This happens on k8s 1.6.1 as well, that replaced old nodes won't get cleaned up, generating large error logs. On CoreOS installation, I did systemctl restart kube-controller-manager. That fixed it.
Ping
This is happening on k8s 1.4.12 as well, restarting kube-controller-manager does not seem to fix it for me. It is rotating through two nodes that no longer exist, so I am hoping the cache will purge itself at some point and they will stop?
Just following up that the old nodes still not cleaned up
will work on a fix for this.
Let's make sure the fix also gets ported back to all affected branches (1.6, 1.5, and 1.4). CC
I was able to reproduce the problem by force deleting a node containing a pod with some volume attached to it. Did anyone run into a different failure mode?
I think that's how the bug surfaced for me. When I force killed my node we had just started running a StatefulSet with an attached persistent volume, and I think there was one running on the very node I killed.
The NodeInformer cache is actually reporting the correct state.

The problem is that inside the node status updater, there is another data structure keeping track of nodes that require a status update (see ActualStateOfWorld.GetVolumesToReportAttached(), called inside UpdateNodeStatuses() in ). The entry for the deleted node is never removed from the data structure; rather, the updater is set to try to update the dead node again next time, which leads to the message being logged every 100ms. The solution is to remove the corresponding node entry once the node is deleted. A fix is on its way.
A fix for 1.5 is on the way (PR )
Will 1.4 be included as well?
Here you go, my apologies.
Thank you very much! :)
To summarize, the fix for this is in 1.7 branch Releases 1.7.0 and greater 1.6 branch Releases 1.6.5 and greater 1.5 branch Releases 1.5.8 and greater (1.5.8 has not yet been cut, CC for ETA of that release). 1.4 branch Releases 1.4.13 and greater (1.4.13 has not yet been cut, CC for ETA of that release).
There is an edge case that the above fix did not handle: see PR is fixing that


Question: } } <del> // TestPriorityQueue_AddUnschedulableIfNotPresent_Async tests scenario when </del> <ins> // TestPriorityQueue_AddUnschedulableIfNotPresent_Backoff tests scenario when </ins> // AddUnschedulableIfNotPresent is called asynchronously pods in and before // current scheduling cycle will be put back to activeQueue if we were trying // to schedule them when we received move request. <del> func TestPriorityQueue_AddUnschedulableIfNotPresent_Async(t *testing.T) { </del> <ins> func TestPriorityQueue_AddUnschedulableIfNotPresent_Backoff(t *testing.T) { </ins> q := NewPriorityQueue(nil) totalNum := 10 expectedPods := make([]v1.Pod, 0, totalNum)

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: Command:  []string{"tar", "cf", "-", t.src.File.String()}, Executor: &exec.DefaultRemoteExecutor{}, } <del> if t.o.MaxTries > 0 { </del> <ins> if t.o.MaxTries != 0 { </ins> options.Command = []string{"sh", "-c", fmt.Sprintf("tar cf - %s | tail -c+%d", t.src.File, n)} }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Trying to copy a large file from a container via kubectl cp with the new flag results in the following error: I executed the following command: EDIT: The command executes and the file is copied. However, the resulting file is somehow corrupted: However, the files have exactly the same size locally and on the pod: Also, the command exits with non-zero code and thus scripts may stop or abort at this point. What you expected to happen: The large file to be downloaded from the container completely and unharmed. Exit code 0 only if copy was successful How to reproduce it (as minimally and precisely as possible): Add a large file (mine was ~150M) to any location in the pod (preferrably a tar file). Execute kubectl cp to download the large file and use the flag. Try to untar the resulting file Anything else we need to know?: Related to Issue . I only experience this issue if I see kubectl doing at least one retry. Environment: Kubernetes version (use kubectl version): server v1.20.12, client v1.23.1 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. uname -a): Install tools: Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
tar: Removing leading `/' from member names so cancel first "/" from "" like this: k cp --retries=-1
Thanks for the hint. The warning is a and I was just ignoring it in my case. This issue is about the error message and the command exiting with a non-zero exit code.
I just updated the issue description, we see the tar files being corrupted after kubectl did at least one retry
/assign atiratree
Is it correct to start at position bytesRead+1? Or should this be bytesRead instead?

For example, if 1 byte was read (position 0), we would want to resume from position 1, not position 2. Just a thought, I haven't really looked at what initReadFrom does.
It should be correct since the byte start of tail (part of implementaion of initReadFrom) and bytes read match. Meaning that when we read 10 bytes we want to start reading 11th one. The actual problem was that we were running a wrong command when -1 retries is used: please see


Question: { Name: logVolume, VolumeSource: v1.VolumeSource{ <del> HostPath: &v1.HostPathVolumeSource{Path: path.Dir(tmpLogFile)}, </del> <ins> EmptyDir: &v1.EmptyDirVolumeSource{}, </ins> }, }, {

Leads to this avc: In general, HostPath volumes in tests are bad news for selinux.


Question: } func getDeviceMountPath(volume *reconstructedVolume) (string, error) { <del> volumeAttacher, err := volume.attachablePlugin.NewAttacher() if volumeAttacher == nil || err != nil { return "", err } deviceMountPath, err := volumeAttacher.GetDeviceMountPath(volume.volumeSpec) if err != nil { return "", err } </del> if volume.blockVolumeMapper != nil { <del> deviceMountPath, err = volume.blockVolumeMapper.GetGlobalMapPath(volume.volumeSpec) if err != nil { </del> <ins> // for block volume, we return its global map path return volume.blockVolumeMapper.GetGlobalMapPath(volume.volumeSpec) } else if volume.attachablePlugin != nil { // for filesystem volume, we return its device mount path if the plugin implements AttachableVolumePlugin volumeAttacher, err := volume.attachablePlugin.NewAttacher() if volumeAttacher == nil || err != nil { </ins> return "", err } <ins> return volumeAttacher.GetDeviceMountPath(volume.volumeSpec) } else { return "", fmt.Errorf("blockVolumeMapper or attachablePlugin required") </ins> } <del> return deviceMountPath, nil </del> } func (rc *reconciler) updateStates(volumesNeedUpdate map[v1.UniqueVolumeName]*reconstructedVolume) error {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (block volmode)] disruptive[Disruptive] Should test that pv used in a pod that is force deleted while the kubelet is down cleans up when the kubelet returns. Since when has it been failing: 10/8/19 Testgrid link: https://k8s- Reason for failure: Introduced by but root cause is probably Anything else we need to know:
/assign
/milestone 1.17 /priority important-soon this is consistently failing in release-master-informing job could you please take a look?
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.17 /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17


Question: buf := bytes.NewBuffer([]byte{}) cmd := NewCmdApply(f, buf) <del> cmd.Flags().Set("filename", filenameRC) cmd.Flags().Set("filename", filenameSVC) </del> <ins> if asList { cmd.Flags().Set("filename", filenameRCSVC) } else { cmd.Flags().Set("filename", filenameRC) cmd.Flags().Set("filename", filenameSVC) } </ins> cmd.Flags().Set("output", "name") cmd.Run(cmd, []string{})

kubectl create, delete, etc. can already take a List object. kubectl apply can take the list object when it does creation, but it gives an (ugly) error when it's configuring the objects. Then... expected: Actual:
what's you ?
0-alpha.5
Sounds like a bug. Are you interested in taking a look
sure I will take a look during this weekend.
I think the root cause is , the actually is , in the origin , is not , but after the , this object is nil. Would you like to double check this?
I'll take a look.
if you have make sure about this, please let me know. Thanks :)
Is there any ways we could get this into 1.2? It's actually making our config generation quite painful to have to make a separate file for every object. It looks like got pretty far? would you like to make a PR that we can review?
I don't see how. It could have made 1.2 if a PR had been sent within a few days of the issue being filed.
I unassigned all issues assigned to you. Please only assign issues that are actively being worked on, so that others can know which issues are available to be picked up.
My apologies. If I'm not mistaken, this issue was filed after the code freeze for 1.2. I didn't realize that it was time critical or that it could make the release at that time. However, I should have asked. Sorry for the inconvenience. Thank you. I still plan to address those issues for the next release, as discussed, but will assign each one to myself as I work on it.
I created which fixes this. I'm really hoping we can get this into 1.2 since it's a pretty small, isolated fix but a pretty big bug that is complicating our config generation system. Let me know what else I can do to help.
Thanks for the PR. We're short on reviewer bandwidth at the moment. I assigned it to Clayton for review or delegation.


Question: } } <ins> // Regularly send heartbeat event to APIServer so that the cluster doesn't enter fullyDisruption mode. // TODO(Huang-Wei): use "NodeDisruptionExclusion" feature to simply the below logic when it's beta. </ins> for i := 0; i < nodeCount; i++ { <del> // Stop the neededNode's heartbeat goroutine. if neededNode.Name == fmt.Sprintf("node-%d", i) { heartbeatChans[i] <- struct{}{} break </del> <ins> var conditions []v1.NodeCondition // If current node is not <neededNode> if neededNode.Name != nodes[i].Name { conditions = []v1.NodeCondition{ { Type:   v1.NodeReady, Status: v1.ConditionTrue, }, } } else { c, err := nodeReadyStatus(test.nodeConditions) if err != nil { t.Error(err) } // Need to distinguish NodeReady/False and NodeReady/Unknown. // If we try to update the node with condition NotReady/False, i.e. expect a NotReady:NoExecute taint // we need to keep sending the update event to keep it alive, rather than just sending once. if c == v1.ConditionFalse { conditions = test.nodeConditions } else if c == v1.ConditionUnknown { // If it's expected to update the node with condition NotReady/Unknown, // i.e. expect a Unreachable:NoExecute taint, // we need to only send the update event once to simulate the network unreachable scenario. nodeCopy := nodeCopyWithConditions(nodes[i], test.nodeConditions) if err := updateNodeStatus(cs, nodeCopy); err != nil && !apierrors.IsNotFound(err) { t.Errorf("Cannot update node: %v", err) } continue } </ins> } <del> } neededNode.Status.Conditions = test.nodeConditions // Update node condition. err = updateNodeStatus(cs, neededNode) if err != nil { t.Fatalf("Cannot update node: %v", err) </del> <ins> // Keeping sending NodeReady/True or NodeReady/False events. go func(i int) { for { select { case <-context.ctx.Done(): return case <-time.Tick(heartbeatInternal): nodeCopy := nodeCopyWithConditions(nodes[i], conditions) if err := updateNodeStatus(cs, nodeCopy); err != nil && !apierrors.IsNotFound(err) { t.Errorf("Cannot update node: %v", err) } } } }(i) </ins> } if err := waitForNodeTaints(cs, neededNode, test.nodeTaints); err != nil {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> The original file is https://github.com/kubernetes-csi/external-resizer/blob/<version>/deploy/kubernetes/rbac.yaml </del> <ins> The original file is https://github.com/kubernetes-csi/external-resizer/blob/VERSION/deploy/kubernetes/rbac.yaml </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In below README files, URLs are written. But these URL include version name(master, release-1.0, etc...) which are enclosed in <and not shown in md files. What you expected to happen: URL will be shown correctly.
/sig storage
/remove-kind bug /kind documentation
can I take it up, as there is no activity for some days?
I already posted PR, thanks.


Question: klog.V(4).Infof("Trying to Apply Policies for service %s", spew.Sdump(svcInfo)) var hnsLoadBalancer *loadBalancerInfo var sourceVip = proxier.sourceVip <del> if containsPublicIP { </del> <ins> if containsPublicIP || containsNodeIP { </ins> sourceVip = proxier.nodeIP.String() } hnsLoadBalancer, err := hns.getLoadBalancer(

What happened: Executing inside a Windows Pod, but get the following msg:  What you expected to happen: Executing inside a Windows Pod, could view: How to reproduce it (as minimally and precisely as possible): Flannel mode to create a Cluster with one control plane (via RKE) and one Windows Node (via ) into the Windows Pod is fine is fine , but get timeout error Anything else we need to know?: Windows workload yaml:Nodes info:  Content of from :More researching:Deploy an Nginx workload with host port exposed, and then create an Endpoint and Service for it, finally access the vip from Windows inside:   Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): Linux: Windows: Kernel (e.g. ): Linux: Windows: Install tools: Control plane created via RKE, Windows node created by Kuberentes guide Network plugin and version (if this is a network-related bug): Flannel VxLan
/sig windows
More information If we edit the default kubernetes endpoint to point to the node’s public ip, then works
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
can you provide the Kubernetes version for the windows node? I see platform as linux in your post
Hi , it happens in 1.14, 1.15 and 1.16. You sent us a commit to fix it before. It works well. Hope we can get it fixed in upstream as well.
In which version was it fixed?

The k8s version I use is: windows Server core version: The problem still appears
The fix for the issue above is included in 1.18.1 to help further
But it doesn't work in 1.18.1
Verified in K8s 1.18. It only works in some particular Windows versions. I guess it is a Windows (OS) issue. works. doesn't work. If you are using AWS, please use instead of the current default one . CC
I do n’t think it should be a system problem, I am using Microsoft Windows Server version 1909 [OS build .778]
: we have identified a potential issue in this area last week. In order to make sure we are talking about the same issue, would it be possible to collect some logs using the instructions below, please? an elevated powershell Window on the host and run: wget -o collectlogs.ps1 the same command line, run .collectlogs.ps1 share the resulting logs folder
Here is a link to the issue I mentioned in my previous comment: Though the issue description is different, the end result is the same: is not accessible from within the POD
any news about that? I'm also running into this issue with newer Windows Versions and can't stay on build .1039
: Two fixes were merged to the master branch a couple of weeks ago. kubernetes/kubernetes kubernetes/kubernetes AFAICT there is no kubernetes releases yet that include these fixes, but I am expecting the next v1.19.x to include these.
and FYI.


Question: return cs.CoreV1().Pods(ns).Create(context.TODO(), pod, metav1.CreateOptions{}) } <del> func checkPodLogs(ctx context.Context, getCalls func(ctx context.Context) ([]drivers.MockCSICall, error), pod *v1.Pod, expectPodInfo, ephemeralVolume, csiInlineVolumesEnabled, csiServiceAccountTokenEnabled bool, expectedNumNodePublish int) error { </del> <ins> // checkNodePublishVolume goes through all calls to the mock driver and checks that at least one NodePublishVolume call had expected attributes. // If a matched call is found but it has unexpected attributes, checkNodePublishVolume skips it and continues searching. func checkNodePublishVolume(ctx context.Context, getCalls func(ctx context.Context) ([]drivers.MockCSICall, error), pod *v1.Pod, expectPodInfo, ephemeralVolume, csiInlineVolumesEnabled, csiServiceAccountTokenEnabled bool) error { </ins> expectedAttributes := map[string]string{} <ins> unexpectedAttributeKeys := sets.New[string]() </ins> if expectPodInfo { expectedAttributes["csi.storage.k8s.io/pod.name"] = pod.Name expectedAttributes["csi.storage.k8s.io/pod.namespace"] = pod.Namespace expectedAttributes["csi.storage.k8s.io/pod.uid"] = string(pod.UID) expectedAttributes["csi.storage.k8s.io/serviceAccount.name"] = "default" <ins> } else { unexpectedAttributeKeys.Insert("csi.storage.k8s.io/pod.name") unexpectedAttributeKeys.Insert("csi.storage.k8s.io/pod.namespace") unexpectedAttributeKeys.Insert("csi.storage.k8s.io/pod.uid") unexpectedAttributeKeys.Insert("csi.storage.k8s.io/serviceAccount.name") </ins> } if csiInlineVolumesEnabled { // This is only passed in 1.15 when the CSIInlineVolume feature gate is set. expectedAttributes["csi.storage.k8s.io/ephemeral"] = strconv.FormatBool(ephemeralVolume) <ins> } else { unexpectedAttributeKeys.Insert("csi.storage.k8s.io/ephemeral") </ins> } if csiServiceAccountTokenEnabled { expectedAttributes["csi.storage.k8s.io/serviceAccount.tokens"] = "<nonempty>" <ins> } else { unexpectedAttributeKeys.Insert("csi.storage.k8s.io/serviceAccount.tokens") </ins> } <del> // Find NodePublish in the GRPC calls. foundAttributes := sets.NewString() numNodePublishVolume := 0 numNodeUnpublishVolume := 0 </del> calls, err := getCalls(ctx) if err != nil { return err } <ins> var volumeContexts []map[string]string </ins> for _, call := range calls { <del> switch call.Method { case "NodePublishVolume": numNodePublishVolume++ if numNodePublishVolume == expectedNumNodePublish { // Check that NodePublish had expected attributes for last of expected volume for k, v := range expectedAttributes { vv, found := call.Request.VolumeContext[k] if found && (v == vv || (v == "<nonempty>" && len(vv) != 0)) { foundAttributes.Insert(k) framework.Logf("Found volume attribute %s: %s", k, vv) } } </del> <ins> if call.Method != "NodePublishVolume" { continue } volumeCtx := call.Request.VolumeContext // Check that NodePublish had expected attributes foundAttributes := sets.NewString() for k, v := range expectedAttributes { vv, found := volumeCtx[k] if found && (v == vv || (v == "<nonempty>" && len(vv) != 0)) { foundAttributes.Insert(k) </ins> } <del> case "NodeUnpublishVolume": framework.Logf("Found NodeUnpublishVolume: %+v", call) numNodeUnpublishVolume++ </del> } <del> } if numNodePublishVolume < expectedNumNodePublish { return fmt.Errorf("NodePublish should be called at least %d", expectedNumNodePublish) } </del> <ins> if foundAttributes.Len() != len(expectedAttributes) { framework.Logf("Skipping the NodePublishVolume call: expected attribute %+v, got %+v", format.Object(expectedAttributes, 1), format.Object(volumeCtx, 1)) continue } </ins> <del> if numNodeUnpublishVolume == 0 { return fmt.Errorf("NodeUnpublish was never called") </del> <ins> // Check that NodePublish had no unexpected attributes unexpectedAttributes := make(map[string]string) for k := range volumeCtx { if unexpectedAttributeKeys.Has(k) { unexpectedAttributes[k] = volumeCtx[k] } } if len(unexpectedAttributes) != 0 { framework.Logf("Skipping the NodePublishVolume call because it contains unexpected attributes %+v", format.Object(unexpectedAttributes, 1)) continue } return nil </ins> } <del> if foundAttributes.Len() != len(expectedAttributes) { return fmt.Errorf("number of found volume attributes does not match, expected %d, got %d", len(expectedAttributes), foundAttributes.Len()) </del> <ins> if len(volumeContexts) == 0 { return fmt.Errorf("NodePublishVolume was never called") </ins> } <del> return nil </del> <ins> return fmt.Errorf("NodePublishVolume was called %d times, but no call had expected attributes %s or calls have unwanted attributes key %+v", len(volumeContexts), format.Object(expectedAttributes, 1), unexpectedAttributeKeys.UnsortedList()) </ins> } // createFSGroupRequestPreHook creates a hook that records the fsGroup passed in

1 ci-kubernetes-e2e-gci-gce 1 ci-cos-containerd-e2e-cos-gce in the master blocking test grid 1 ci-kubernetes-e2e-gci-gce-network-proxy-grpc 1 ci-kubernetes-e2e-gci-gce-proto Kubernetes e2e suite.[It] [sig-storage] CSI Mock workload info CSI PodInfoOnMount Update should be passed when update from false to true 12/14 code thaw after 1.29.0 release was merged at the time and is related to PodInfoOnMount. /cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
testCase: podInfoOnMount should be passed when update from false to true kubelet log: csi-mockplugin-0-mock log: the function in the test case is not working as expected. expect log at Line 2, but actual log is at Line 3.
is called twice for the same volume sometime even though the volume is already published successfully. Not only this issue, but also other issues like I wonder if it is a potential bug or a normal behavior. cc what do you think?
good catch. I think that any test that depends on exact index of NodePublish call will be flaky, sometimes Kubelet has to retry NodePublish. It would be better to check for NodePublish parameter changes. For example, test should check there were 1 or more NodePublish calls with no pod info and then 1 or more NodePublish calls with pod info afterwards.