Question: if !addr.Equal(expected) { t.Errorf("Unexpected mismatch, expected: %v, got: %v", expected, addr) } <del> fake.SetLocalAddresses("kube-ipvs0", "4.3.2.1") </del> <ins> fake.SetLocalAddresses("kube-ipvs0", "1.2.3.4", "4.3.2.1") </ins> addr, _ = fake.GetAllLocalAddresses() expected = sets.New("1.2.3.4", "4.3.2.1") if !addr.Equal(expected) { t.Errorf("Unexpected mismatch, expected: %v, got: %v", expected, addr) } <ins> addr, _ = fake.GetAllLocalAddressesExcept("kube-ipvs0") expected = sets.New("1.2.3.4") if !addr.Equal(expected) { t.Errorf("Unexpected mismatch, expected: %v, got: %v", expected, addr) } </ins> }

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: kube::test::get_object_assert daemonsets "{{range.items}}{{$id_field}}:{{end}}" '' # Command # Create a DaemonSet (revision 1) <del> kubectl apply -f hack/testdata/rollingupdate-daemonset.yaml "${kube_flags[@]}" </del> <ins> kubectl apply -f hack/testdata/rollingupdate-daemonset.yaml --record "${kube_flags[@]}" kube::test::get_object_assert controllerrevisions "{{range.items}}{{$annotations_field}}:{{end}}" ".*rollingupdate-daemonset.yaml --record.*" </ins> # Rollback to revision 1 - should be no-op kubectl rollout undo daemonset --to-revision=1 "${kube_flags[@]}" kube::test::get_object_assert daemonset "{{range.items}}{{$daemonset_image_field0}}:{{end}}" "${IMAGE_DAEMONSET_R1}:" kube::test::get_object_assert daemonset "{{range.items}}{{$container_len}}{{end}}" "1" # Update the DaemonSet (revision 2) <del> kubectl apply -f hack/testdata/rollingupdate-daemonset-rv2.yaml "${kube_flags[@]}" </del> <ins> kubectl apply -f hack/testdata/rollingupdate-daemonset-rv2.yaml --record "${kube_flags[@]}" </ins> kube::test::wait_object_assert daemonset "{{range.items}}{{$daemonset_image_field0}}:{{end}}" "${IMAGE_DAEMONSET_R2}:" kube::test::wait_object_assert daemonset "{{range.items}}{{$daemonset_image_field1}}:{{end}}" "${IMAGE_DAEMONSET_R2_2}:" kube::test::get_object_assert daemonset "{{range.items}}{{$container_len}}{{end}}" "2" <ins> kube::test::wait_object_assert controllerrevisions "{{range.items}}{{$annotations_field}}:{{end}}" ".*rollingupdate-daemonset-rv2.yaml --record.*" </ins> # Rollback to revision 1 with dry-run - should be no-op kubectl rollout undo daemonset --dry-run=true "${kube_flags[@]}" kube::test::get_object_assert daemonset "{{range.items}}{{$daemonset_image_field0}}:{{end}}" "${IMAGE_DAEMONSET_R2}:"

What happened: What you expected to happen: DS history revision 2 should have change cause from the second command "kubectl apply -f ds-2.yaml --record", not the first /assign


Question: if _, err := fmt.Fprint(w, AppendAllLabels(options.ShowLabels, ingress.Labels)); err != nil { return err } <del> // Lay out all the rules on separate lines if use wide output. // TODO(AdoHe): improve ingress output extraLinePrefix := "" if options.WithNamespace { extraLinePrefix = "t" } for _, rules := range hostRules { if rules.HTTP == nil { continue } _, err := fmt.Fprintf(w, "%st%vt", extraLinePrefix, rules.Host) if err != nil { return err } if _, err := fmt.Fprint(w, AppendLabelTabs(options.ColumnLabels)); err != nil { return err } for _, rule := range rules.HTTP.Paths { _, err := fmt.Fprintf(w, "%st%vt%v", extraLinePrefix, rule.Path, backendStringer(&rule.Backend)) if err != nil { return err } if _, err := fmt.Fprint(w, AppendLabelTabs(options.ColumnLabels)); err != nil { return err } } } </del> return nil }

We should show a nicely formatted map, with tls and ports. This 50% works. Today we have: Which works well enough, but printing a single Ingress involves hand managing tabs and newlines till the entire Ingress is printed, i.e: So when we start mixing different Ingress types, we get jarring output: This is because the columns are formatted per the top ingress and don't match the bottom one. The tabbed writer used by kubectl to align columns gets confused on the whole.
you can assign this to me, will fix this next-candidate.
thanks!
Sounds like we should keep it to a single line () and move everything else to describe. Makes this bug easier.
yeah, we will move this to a single line, this is the principle.
was this finished for 1.3?
the code is already finished on my local branch. I will open a PR as I confirmed it.
what do you think if print ingress like this: Also the original resource is:
That would fall right off the screen. I think we have to stick to just printing the ips in , and everything else in . You can print hoststs alone, and optimize for small ingresses by doing something like we do for endpoints (show a small number of characters then do "..."). The problem with trying to print everything on one line is you can't represent the fanout cleanly, ie you might do: But with multiple hosts: Doesn't work, because /bar and secret2 only applies to
I agree with this, maybe we should just remove these rules from get ingress output or just leave host here.
SGTM are you going to send a pr?
yes, ready to.


Question: PriorityClassName: "system-cluster-critical", }, }, <ins> // pod[5]: mirror Pod with a system priority class name { ObjectMeta: metav1.ObjectMeta{ Name:        "mirror-pod-w-system-priority", Namespace:   "namespace", Annotations: map[string]string{api.MirrorPodAnnotationKey: ""}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name: containerName, }, }, PriorityClassName: "system-cluster-critical", }, }, // pod[6]: mirror Pod with integer value of priority { ObjectMeta: metav1.ObjectMeta{ Name:        "mirror-pod-w-integer-priority", Namespace:   "namespace", Annotations: map[string]string{api.MirrorPodAnnotationKey: ""}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name: containerName, }, }, PriorityClassName: "default1", Priority:          &intPriority, }, }, </ins> } // Enable PodPriority feature gate. utilfeature.DefaultFeatureGate.Set(fmt.Sprintf("%s=true", features.PodPriority))

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Priority admission controller of mirror pods. This is an error and should be removed. What you expected to happen: Integer value of Priority should be populated by the admission controller. How to reproduce it (as minimally and precisely as possible): Create a static pod with PriorityClassName set. The created mirror pod will not get the integer value of priority. /sig scheduling xref
what is the kubelet expected to do with a low priority static pod if it gets evicted?
also doesn't this mean a user can claim their pod is a mirror pod via an annotation and get it admitted with any priority they want?
knows better about kubelet evictions, but in my opinion a static low priority pod should be evicted similar to other pods. Whether it makes sense for static pods to be low priority or not is a relevant question. I would say it doesn't make sense, but one might find a reason to have static low priority pod. Not really. This check effectively bypasses integer resolution of priority causing the priority of all those mirror pods to be considered zero. This if statement is wrong and should be removed.
It also bypasses the check that prevents the pod creator from setting an integer priority themselves. In any case, agree removing the bypass is good
Static pods that fail for any reason (eviction is just one) are not re-run. To avoid evictions entirely as a static pod, users can use the critical pod annotation.
, it seems we just log error if creating mirror pod failed; refer to . Does it make sense?
I am not sure I understand the question entirely. The mirror pod exists only to show the status of the static pod. If the mirror pod doesn't exist, the static pod should still function correctly. Since that is in , it will be retried periodically.
The scheduler dependent on mirror pods for resource usage :). If creating mirror pod failed (e.g. wrong priority class name), there'll be ping-pong between scheduler and kubelet: scheduler send pod to node, kubelet reject it. But we can-t just return error, kubelet can start in standalone mode.
However critical pod annotation is ignored if pod priorities are in use.

This is not the case. See in your issue.
However critical pod annotation is ignored if pod priorities are in use. the intent of pod priorities was to replace the critical pod annotation with something more principled. is that not how it was implemented?
Priority replaces most of the functionality that the critical pod annotation provides. However, the critical pod annotation is still useful for static pods, as it can prevent kubelet evictions. But for all other use-cases, I agree that priority is a much better way to manage which workloads are run, and which remain pending. I am not sure if the scheduler translates a "critical" pod to a "system-priority" pod for scheduling purposes, but the kubelet still prevents the eviction of static critical pods as an escape hatch for important static pods which cannot tolerate kubelet evictions.
Why would that not be modeled as a system priority? I expected critical pod annotations to not be present once the priority feature was completed.
I agree with this. do you think we should not evict static pods which are system priority, so that even users of static pods can move from using the critical pod annotation to use pod priority?
I think Kubelet should not evict static system pods, but even with the current logic of eviction, if these Pods do not use more resources than requested, kubelet never evicts them as their priority is higher than other pods.
I think as a best practice users that write static pod manifests should apply a system priority to prevent eviction, but I do not think the kubelet should do anything magical about that source.


Question: // If ignoreExistErr is set to true, then the -exist option of ipset will be specified, ipset ignores the error // otherwise raised when the same set (setname and create parameters are identical) already exists. func (runner *runner) createSet(set *IPSet, ignoreExistErr bool) error { <del> args := []string{"create", set.Name, string(set.SetType), "comment"} </del> <ins> args := []string{"create", set.Name, string(set.SetType)} </ins> if set.SetType == HashIPPortIP || set.SetType == HashIPPort { args = append(args, "family", set.HashFamily,

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area kube-proxy /area ipvs /assign I'll send a fixing pull request tomorrow. What happened: On centos 7.2 with ipset 6.29, kube-proxy in ipvs mode failed to create ipset sets. After some dig, it turns out to be problem of ipset on centos/rh. See I tried ipset 6.29 on ubuntu, it's OK. But when I tried ipset 6.29 on centos, I got errors: What you expected to happen: Since the problematic ipset versions have very large installation base, we could remove ipset commet, or remove ipset commet with problematic ipset version. The problematic ipset also says it support comment by --help. The problem could be detected by a test run of "ipset create foo hash:ip comment". How to reproduce it (as minimally and precisely as possible): Create some service in kubernetes, run kube-proxy with ipvs mode on Centos 7.x node. Anything else we need to know?: Environment: Kubernetes version (use ): commit (actually all commits after ) Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: ipset v6.29, protocol version: 6
could you test it before commit , that's ok?
, yes, I did. It's OK. But I think the we don't need to revert the iptables comments.
if ipset package not installed, will set module or match-set option in rule be weird and now seems not.


Question: tp.disableAttach = true tp.registerDriver = true } <ins> tp.hooks = createExpansionHook(test.simulatedCSIDriverError) </ins> m.init(ctx, tp) ginkgo.DeferCleanup(m.cleanup)

When a pod is quickly deleted and re-created on a node, it may not be necessary to unstage the volume and in which case only node-publish is called for the volume. When this happens - stagepath won't be set for RPC call because volume was not staged even though, volume does have capability. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. Quickly delete and recreate a pod No response master Any No response_ <details</details<details</details<details</details
/sig storage /triage accepted


Question: myConfig.BearerToken = string(test.secretTokenData) roClient := clientset.NewForConfigOrDie(&myConfig) dateBefore := time.Now().UTC().Format(dateFormat) <del> go func() { doServiceAccountAPIRequests(t, roClient, myns, true, true, false) }() doServiceAccountAPIRequests(t, roClient, myns, true, true, false) </del> <ins> var wg sync.WaitGroup concurrency := 5 for i := 0; i < concurrency; i++ { wg.Add(1) go func() { doServiceAccountAPIRequests(t, roClient, myns, true, true, false) wg.Done() }() } wg.Wait() </ins> dateAfter := time.Now().UTC().Format(dateFormat) liveSecret, err := c.CoreV1().Secrets(myns).Get(context.TODO(), test.secretName, metav1.GetOptions{}) if err != nil { t.Fatalf("Could not get secret: %v", err) } <del> if test.expectWarning && len(wh.warnings) != 8 { t.Fatalf("Expect 8 warnings, got %d", len(wh.warnings)) </del> <ins> // doServiceAccountAPIRequests has 4 API requests if test.expectWarning && len(wh.warnings) != 4*concurrency { t.Fatalf("Expect %d warnings, got %d", 4*concurrency, len(wh.warnings)) </ins> } if !test.expectWarning && len(wh.warnings) != 0 { t.Fatalf("Don't expect warnings, got %d", len(wh.warnings))

pull-kubernetes-integration TestLegacyServiceAccountTokenTracking (in top 10 list of flakes - ) 10/25 No response It looks like the number of warnings received is not consistent: No response /sig auth /assign /milestone v1.26 /priority important-soon we should resolve this flake before test freeze
/triage accepted


Question: # Optional: Enable setting flags for kube-apiserver to turn on behavior in active-dev #RUNTIME_CONFIG="" RUNTIME_CONFIG="api/v1" <ins> # Determine extra certificate names for master octets=($(echo "$SERVICE_CLUSTER_IP_RANGE" | sed -e 's|/.*||' -e 's/./ /g')) ((octets[3]+=1)) service_ip=$(echo "${octets[*]}" | sed 's/ /./g') MASTER_EXTRA_SANS="IP:${service_ip},DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.${DNS_DOMAIN},DNS:${MASTER_NAME}" </ins>

On latest Kubernetes version Release 0.20.1 Candidate I get the following when you setup cluster with "./cluster/kube-". This is on Mac OSX with VirtualBox, Vagrant Any ideas, thanks .... ==minion-3:  INFO: Running daemons_running() ==minion-3:  INFO: Salt installed! Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Validating minion-1 Validating minion-2 Validating minion-3 ..... Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused kubernetes user1$ get nodes error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
the same for the first time when i ran v0.20.1 (the published one)
here is the installation log on my core i5-5200U ubuntu 14.04 3.16 kernel this is that console says
See my comment on
I think is probably right. There's a 0.20.2 brewing already (and we should have it out later today), so hang tight.
I just tried this also with Release 0.20.2 Candidate on Ubuntu 14.04 with VirtualBox, Vagrant and I am seeing:
- Is there someone on your team that could take a look?
Honestly no, everyone is busy, and working on either cluster or network related issue. But I talked to briefly, he kindly agree to take a look. The cloud provider here is Vagrant, if you could reproduce the issue, that would be huge help. Feel free to re-assign it to me if you are blocked. Thanks!
cc/ since he looked into vagrant issue lately. cc/ on Vagrant.

+1, 0.20.2 still has this issue
+1
The fix for this issue is this PR:
Assigning issue to me since I am focused today on closing out remaining Vagrant setup issues.
Sigh, I'm sorry. So busy working through unassigned PRs I forgot to work through my own queue.
I was running into the same issue and cherry picked your changes in , but still have the same issue. Prior to those changes the kubelet never started on the master. With the changes it does start, but I still get: It looks like also needed before vagrant works correctly.
Yep, I picked and then merged and it started to work again... just my 2 cents
this error is gone in 0.21.0 rc ()
I am using Mac OSX 10.10.5 / VMWare Fusion / bento/fedora-21/ Vagrant 1.7.4 / and built kubernetes from master (GitVersion:"v1.1.0-alpha.1.687+") and I am seeing this issue when I run ./cluster/kube- Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Connection to 192.168.100.177 closed. ..Validating minion-1 ............. Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
I'm running into a similar issue with a Kubernetes cluster on AWS. I set it up last week and now can't connect to the master node: oamasood kubectl cluster-info error: couldn't read version from server: Get https://11.11.111.111/api: dial tcp 11.11.111.111:443: connection refused Any help would be appreciated.
I find that if you wait for the master and minion to calm down on activity after a bit (~5 minutes) the kubectl commands work, however the problem is I don't see any nodes running.

I have re-installed virtual box, made sure my .kube directory is clean and I am now pulling just the stock vagrant box cube fedora 21 and kubernetes 1.0.4 release. I do not see any docker processes running on my minion. Other than that it seems that the cluster is running: $ get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok nil scheduler Healthy ok nil etcd-0 Healthy {"health": "true"} nil $ cluster-info Kubernetes master is running at https://10.245.1.2 KubeDNS is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-dns KubeUI is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-ui yggdrasil:kubernetes jonathantronson$ vagrant ssh master Last login: Wed Sep 16 23:49:51 2015 from 10.0.2.2 [vagrant ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES "/bin/sh -c '/usr/loc" 50 minutes ago Up 50 minutes k8skube-apiserver.2abef58bkube-apiserver-kubernetes-masterkube-system496e220b4704d4b390883b4b90aa66354e41e61f "/bin/sh -c '/usr/loc" 50 minutes ago Up 50 minutes k8setcd-container.fa2ab1d9etcd-server-kubernetes-masterkube-system6ef1b07678064a7edcdbec6f0570e91759d753f9 "/bin/sh -c '/usr/loc" 53 minutes ago Up 53 minutes k8skube-scheduler.ea78c200kube-scheduler-kubernetes-masterkube-system3d1ced0542467b566546021cb74d9bdf9550be6a "/bin/sh -c '/usr/loc" 53 minutes ago Up 53 minutes k8skube-controller-manager.fd0f0b82kube-controller-manager-kubernetes-masterkube-system2f59d05ecb2212d0379dc35dbe40d5216292ac06 "/pause" 54 minutes ago Up 54 minutes k8sPOD.

e4cc795kube-scheduler-kubernetes-masterkube-system3d1ced0542467b566546021cb74d9bdf62e6c529 "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795kube-controller-manager-kubernetes-masterkube-system2f59d05ecb2212d0379dc35dbe40d52109062f04 "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795etcd-server-kubernetes-masterkube-system6ef1b07678064a7edcdbec6f0570e9173d73246b "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795kube-apiserver-kubernetes-masterkube-system496e220b4704d4b390883b4b90aa6635c1bfe2c3 [vagrant ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
The only errors in my minion salt log are: 2015-09-16 23:40:22,049 [][ERROR ][2668] Command 'systemctl status docker.service' failed with return code: 3 2015-09-16 23:40:22,049 [][ERROR ][2668] output: * docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled) Active: inactive (dead) Docs:
Hi, I am running the latest version of kubernetes tar ball (v1.1.3) and facing issues with ./kube- I was unable to fix the issue. Any help would be appreciated. cluster "vagrant" set. user "vagrant" set. context "vagrant" set. switched to context "vagrant". Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Connection to 127.0.0.1 closed.

Validating minion-1 Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused Thank You
Mac 10.11.2 installing kubernetes 1.1.3 got same issue: I think the fix didn't catch the root cause.
same issues on 1.1.7 cluster "vagrant" set. user "vagrant" set. context "vagrant" set. switched to context "vagrant". Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master ^ to 127.0.0.1 closed. Validating minion-1 ...^ Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
when I rerun the cluster/kube-, the error still exist, and can't get nodes/services etc status. The first time, it works. weird.
can u help to take a look this issues? thanks
- can you open a new issue and describe what host environment you are using? Are you testing from master or a branch? cc me on the issue.
I used released version 1.1.7. btw, I tried several times. even at the last step failed. I can run get pods/services etcs. seems during verify stage. it will search local kubectl files? I am not sure. as on my Mac , there have some mismatch version kubectl file.
I experience the very same issue on AWS and, usually, I can't connect to my cluster anymore the day after I had set it up - which is a huge blocker. Here's the output from :
I expreience the same issue on Ubuntu.
The issue disappeared for me with the following version:
I encounter the same problem,can you give me a detail configuration of kuberenetes?

ubuntu /opt/bin/kubectl version Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.8", GitCommit:"", GitTreeState:"clean"} error: couldn't read version from server: Get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused


Question: matchingAntiAffinityTerms: matchingTerms, } for predicateName, precomputeFunc := range predicatePrecomputations { <del> glog.V(4).Info("Precompute: %v", predicateName) </del> <ins> glog.V(10).Info("Precompute: %v", predicateName) </ins> precomputeFunc(predicateMetadata) } return predicateMetadata

Problem There are at least some examples of over logging in the scheduler. Solution Go through glog.Info and make sure we are consistently managing logs. At least one example to solve in an easy PR has shared with us.. Ill confirm this is happening still in upstream kube and fix as well as audit the others.
part of
yeah lets de-dupe the tickets, we have 3+ issues on this now.. closing this one. just submit the PRs and reference one of the other tickets.


Question: func (m *podContainerManagerImpl) Destroy(podCgroup CgroupName) error { // Try killing all the processes attached to the pod cgroup if err := m.tryKillingCgroupProcesses(podCgroup); err != nil { <del> klog.V(3).Infof("failed to kill all the processes attached to the %v cgroups", podCgroup) </del> <ins> klog.Warningf("failed to kill all the processes attached to the %v cgroups", podCgroup) </ins> return fmt.Errorf("failed to kill all the processes attached to the %v cgroups : %v", podCgroup, err) }

What happened: Housekeeping sync loop finds an orphaned pod and tries to clean it up. The kubepod slice, as seen below, is not removed by with no indication as to why. This leads to the docker scope timing out and entering a failed state. The returned error(s) from runc are swallowed making debugging difficult. What you expected to happen: As per other areas of the sync loop, useful errors are logged at debug level due to the async calls. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: N/A Environment: Kubernetes version (use ): master
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Have you seen the comment above pcm.Destroy() call ? Did you have a chance to look at kubelet log to see if the above was true ?
/remove-lifecycle stale
This would give us some clue about the error:
yes, definitely looked! In this specific situation, the pods are never retried. What we see happen with these pods: ADD -RECONCILE -Started Volumes -Created slice -No sandbox for pod -Started libcontainer -ContainerStarted -ContainerStarted -RECONSILE -ContainerDied -RECONCILE -Unmounts -Killing unwanted pod -Orphaned removing pod cgroups -container kill failed because of 'container not found' or 'no such process' -cleanup: failed to unmount secrets -ContainerDied -Container garbage collection succeeded -SyncLoop (SYNC): 141 pods (including this one) -stopping timed out. Killing.

-Stopped libcontainer -Removed slice -DELETE -REMOVE -err: pod not found Destroy all cgroups of the pod that should not be running, by first killing all the attached processes to these cgroups. We ignore errors thrown by the method, as the housekeeping loop would again try to delete these unwanted pod cgroups Despite running, the slice is not removed and we do not log or indicate why. Subsequently, the pod is NOT determined to be either an active pod or for some reason not in the cgroup list that is passed into kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) during housekeeping loops.

A more detailed look, which I believe is accurate ;): housekeepingCh -HandlePodCleanups -cgroupPods=UID+CGroupName=GetAllPodsFromCgroups -allPods=GetPodsAndMirrorPods -activePods=filterOutTerminatedPods -cleanupOrphanedPodCgroups I do not think desiredPods and runningPods are relevant here.. So either: a) the pod is not found and to allPods (seems unlikely) but we can't know for sure.. b) the pod is incorrectly determined to be active via activePods which again we can not determine.. Any pod that is not Failed, Succeeded or has a DeletionTimestamp set and a status of Terminated or Waiting - anything else returns false and is deemed active c) the pod cgroup is not found in m.subsystems.MountPoints and to cgroupPods - unlikely, but could be a bug but the root of this issue is the fact that all of the scopes and slices exist and keep building up until things start failing... GetAllPodsFromCgroups could use some logging... Maybe a V 5 log that prints out foundPods at the end... From there, they're passed into the cleanup: kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) 1) All activePod UIDs are put into a podSet 2) A range loop of uid and cgroup name from cgroupPods looks to see if the uid from cgroupPods also exists in the podSet (active pods) 3) If the uid from cgroupPods exists in the podSet we skip out of the loop 4) If it doesn't, the pod is orphaned (not active w/ cgroups on the filesystem) and we call pcm.Destroy(val) on the cgroup name. Doesn't make sense why pcm.Destroy(val) ignores errors... it would be helpful!

tryKillingCgroupProcesses logs any error and also returns the an error which is always swallowed.. we don't see any issues here More problematic, the call to cgroupManager.Destroy only returns an error which would be swallowed... Could probably use a similar level 3 log here like what you have for tryKillingCgroupProcesses because as you can see, were totally blind here...
your change is of a similar nature to what I proposed in my PR for this issue, just a level higher. I think there are a few areas noted above the could help further debug this.. Still haven't been able to get my PR merged with additional logging.
The following code was relatively new (compared to when you logged this issue):
Indeed... I looked into it a while ago and those changes are new via: - Could definitely make a different if the issue is related to pod getting filtered out from activePods when they have not been fully cleaned up yet on the subsequent housekeeping syncloop runs.
I think is promising, but I still think it makes sense to add some logging here. Do you want want to proceed with my open PR, your suggestion in or both?
You can proceed with your PR.


Question: cacheddiscovery "k8s.io/client-go/discovery/cached" "k8s.io/client-go/dynamic" clientset "k8s.io/client-go/kubernetes" <ins> restclient "k8s.io/client-go/rest" </ins> csiclientset "k8s.io/csi-api/pkg/client/clientset/versioned" "k8s.io/kubernetes/pkg/controller" cloudcontroller "k8s.io/kubernetes/pkg/controller/cloud"

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: errExpected:        context.Canceled, }, { <ins> name:               "context cancelled and immediate after 5 attempts", context:            defaultContext, callback:           defaultCallback, immediate:          true, cancelContextAfter: 5, attemptsExpected:   5, errExpected:        context.Canceled, }, { name:             "context at deadline and immediate 1 attempt expected", context:          deadlinedContext, callback:         defaultCallback, immediate:        true, attemptsExpected: 1, errExpected:      context.DeadlineExceeded, }, { </ins> name:             "context at deadline no attempts expected", context:          deadlinedContext, callback:         defaultCallback,

Calling wait.PollUntilContextTimeout with immediate=true does not invoke the passed condition function if context has already been cancelled. wait.PollUntilContextTimeout with immediate=true should invoke the passed condition function at least once even if context has already been cancelled as wait.PollUntilContextCancel documentation says: Note that wait.PollUntilContextTimeout wraps wait.PollUntilContextCancel, so the above should be applied to wait.PollUntilContextTimeout as well. This program panics because context ctx.Err is checked before invoking the passed function. The deprecated wait.PollImmediateWithContext does not have this kind of issue. If wait.PollUntilContextTimeout is the replacement of wait.PollImmediateWithContext, it should exhibit the same behavior. k8s v1.27.2 <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
TestloopConditionUntilContextsemantic also has an issue that immediate and sliding are not tested as expected. immediate and sliding are loop variable, but they are not used in the test. The ones that are used are in testcase struct which are always false.
/assign
ok, it seems this would do it
/assign
/triage accepted


Question: assert.Equal(t, newStatus.Message, firstStatus.Message) } <ins> func TestTerminatePodWaiting(t *testing.T) { syncer := newTestManager(&fake.Clientset{}) testPod := getTestPod() t.Logf("update the pod's status to Failed.  TerminatePod should preserve this status update.") firstStatus := getRandomPodStatus() firstStatus.Phase = v1.PodFailed firstStatus.InitContainerStatuses = []v1.ContainerStatus{ {Name: "init-test-1"}, {Name: "init-test-2", State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{Reason: "InitTest", ExitCode: 0}}}, {Name: "init-test-3", State: v1.ContainerState{Waiting: &v1.ContainerStateWaiting{Reason: "InitTest"}}}, } firstStatus.ContainerStatuses = []v1.ContainerStatus{ {Name: "test-1"}, {Name: "test-2", State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{Reason: "Test", ExitCode: 2}}}, {Name: "test-3", State: v1.ContainerState{Waiting: &v1.ContainerStateWaiting{Reason: "Test"}}}, } syncer.SetPodStatus(testPod, firstStatus) t.Logf("set the testPod to a pod with Phase running, to simulate a stale pod") testPod.Status = getRandomPodStatus() testPod.Status.Phase = v1.PodRunning syncer.TerminatePod(testPod) t.Logf("we expect the container statuses to have changed to terminated") newStatus := expectPodStatus(t, syncer, testPod) for i := range newStatus.ContainerStatuses { assert.False(t, newStatus.ContainerStatuses[i].State.Terminated == nil, "expected containers to be terminated") } for i := range newStatus.InitContainerStatuses { assert.False(t, newStatus.InitContainerStatuses[i].State.Terminated == nil, "expected init containers to be terminated") } expectUnknownState := v1.ContainerState{Terminated: &v1.ContainerStateTerminated{Reason: "ContainerStatusUnknown", Message: "The container could not be located when the pod was terminated", ExitCode: 137}} if !reflect.DeepEqual(newStatus.InitContainerStatuses[0].State, expectUnknownState) { t.Errorf("terminated container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.InitContainerStatuses[0].State, expectUnknownState)) } if !reflect.DeepEqual(newStatus.InitContainerStatuses[1].State, firstStatus.InitContainerStatuses[1].State) { t.Errorf("existing terminated container state not preserved: %#v", newStatus.ContainerStatuses) } if !reflect.DeepEqual(newStatus.InitContainerStatuses[2].State, expectUnknownState) { t.Errorf("waiting container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.InitContainerStatuses[2].State, expectUnknownState)) } if !reflect.DeepEqual(newStatus.ContainerStatuses[0].State, expectUnknownState) { t.Errorf("terminated container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.ContainerStatuses[0].State, expectUnknownState)) } if !reflect.DeepEqual(newStatus.ContainerStatuses[1].State, firstStatus.ContainerStatuses[1].State) { t.Errorf("existing terminated container state not preserved: %#v", newStatus.ContainerStatuses) } if !reflect.DeepEqual(newStatus.ContainerStatuses[2].State, expectUnknownState) { t.Errorf("waiting container state not defaulted: %s", diff.ObjectReflectDiff(newStatus.ContainerStatuses[2].State, expectUnknownState)) } t.Logf("we expect the previous status update to be preserved.") assert.Equal(t, newStatus.Phase, firstStatus.Phase) assert.Equal(t, newStatus.Message, firstStatus.Message) } </ins> func TestSetContainerReadiness(t *testing.T) { cID1 := kubecontainer.ContainerID{Type: "test", ID: "1"} cID2 := kubecontainer.ContainerID{Type: "test", ID: "2"}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When a pod is evicted under resource pressure, it goes into an error state. The pod terminates normally. Note that the 130GB file is based on 90% of total disk space - current utilization to hit the default eviction threshold. Full pod spec after going into error state: <details</detailsSee also Eviction tests () are currently broken (), there's a PR up to fix: Kubernetes version (use ): HEAD/1.23.x (kubeletVersion="v1.23.0-alpha.3.47+") Cloud provider or hardware configuration: local-up-cluster OS (e.g: ), Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others: Also reproducible on OpenShift 4.9 (1.23-based) /sig node /cc
mentions this looks very similar to
Confirmed that this is a regression from 1.21. When running the reproducer against the release-1.21 branch, the pod above correctly stays in Evicted status (since pods are mortal). On 1.22, it tries to restart, entering either an Error or ContainerCreating state. /kind regression
/assign
/triage accepted
From : is the first bad commit This was
Seems likely that wasn't actually fixed. The broken eviction tests made it much harder to verify.
Attaching logs from HEAD from reproducer: v=4 Let me also try to get a repro where it goes into Error
Logs from HEAD for reproducer when pod goes into Error: v=4
Roughly, kubelet syncTerminatingPod sends "Failed" to status manager (which looks like it generates the correct apiserver patch for phase running -failed), then somehow by the time Kubelet syncTerminatedPod is called it sends "Running but with errors" to status manager, and status manager allows that transition (internally) and then sends it to the apiserver. So the pod goes Running -Failed -Running which should never happen (we "lost" that the pod was stopped).

Bug 1: pod patch status probably shouldn't allow terminal -nonterminal phase transitions (didn't we fix that??) Bug 2: pod worker is supposed to be getting most up to date status from status manager before calling syncTerminatedPod, which should include "Failed", but that isn't happening? thus the status manager then sends a "non-terminal state" (shouldn't ever happen) Roughly, figure out why status passed to syncTerminatedPod is older than what was sent by syncTerminatingPod, then figure out why status manager allowed that transition back to running, then figure out why apiserver didn't reject.
Staring at logs, I've figured out where the illegal phase transition is happening: If the container failed to properly terminate and generate a status saying as much, generateAPIPodStatus won't have any hints available to it that the pod died because it was killed. (This is happening in the reproducer case because the container won't exit before the termination grace period, and needs to be force killed.) Thus, it will assume that since the container used to be running, its current status is running. This results in an illegal phase transition.
From what I can tell, the issue is that is being passed a pod that still has running containers. From kubelet logs: which gets set as part of . It looks like that will always run so long as succeeds. Thus, I feel like syncTerminatingPod must have the bug: it returns successfully, but the containers of the pod aren't all actually terminated.
/priority critical-urgent based on priority of and the impact of this regression (Kubelet-initiated evictions frequently result in error state pods)
The containers ARE terminated, we just haven't retrieved their status. This is because of a decision to not refresh the container status cache in syncTerminatingPods. So syncTerminatingPod is not a bug.
So then perhaps the right thing to do here would be to refresh the container status cache before syncing another status to the API server in ? Then we can avoid syncing the outdated cached status
Updating from an internal discussion last night - bug 2 isn't actually a bug.

Lantau convinced me we wouldn't refresh the status cache, so what happens is we call generatedAPIStatus with: API status (mostly irrelevant) status manager status to date container cache (but 2s) after The key thing is that phase from runtime status is irrelevant when 1 or 2 are terminal - because runtime has no idea about whether a pod is terminal. So roughly we should be ignoring phase from 3 except when determining when to go from pending to running. The bug here is that we're thinking that this is an "illegal transition" but still copying the latest reason and message (runtime status doesn't get to weigh in on overal terminal phases). We shouldn't need to refresh status at all.
Okay, so there is definitely a bug in what's being synced to the API server for evicted pod statuses right now, possibly for other terminated pods. In 1.21 and earlier, when a pod is evicted, it looks like this: <details</detailsNote that the status field is very minimal, and there are no container statuses. In 1.22+, we attempt to sync container statuses on termination, and sometimes they are wrong (e.g. changing phase from Failed -Running). Here's an example of what an evicted pod could look like on 1.22: <details</details
This appears to be broken even for runOnce pods, tested against latest HEAD (). With spec: After scheduling the pod... And the final pod status: <details</details
Elana identified the issue on status here - previously we NEVER reported container status after an eviction, so the refactor (which correctly ensured container exit codes showed up) was returning containers with state=waiting,lastState=terminated, which is incorrect. generateAPIStatus needs to know the difference between "current container status, waiting for more to show up" and "any status we have is the best we have". So once killPods succeeds in syncTerminatingPods and in syncTerminatedPods, we're no longer "waiting". So generating pod status should correctly summarize the container status as and not inject .


Question: SortBy             string NoHeaders          bool UseProtocolBuffers bool <ins> ShowCapacity       bool </ins> NodeClient      corev1client.CoreV1Interface Printer         *metricsutil.TopCmdPrinter

What happened: We ran the following, and got the following: How/why do we have a node reporting 100% memory usage? (There seems to be plenty of memory on the host as reported by the kernel's statistic. (multiple gigabytes) What you expected to happen: Memory usage can't exceed 100%, no? How to reproduce it (as minimally and precisely as possible): we unfortunately don't know Anything else we need to know?: no swap on these VMs. We're curious what kernel memory statistic goes into computing the total from Kubernetes; it's my understanding that there is various ways to go over 100%, e.g., by summing RSS over several processes. (E.g., shared and resident pages would get double-counted.) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure OS (e.g: ): AKS instance? Kernel (e.g. ): (unsure, since this is Azure AKS; we don't have good access to this piece of data…) Install tools: Azure? Network plugin and version (if this is a network-related bug): N/A / Azure Others: None
(Taking my best shot here, but if this is wrong please freely adjust it.) /sig instrumentation
Other than 101% reporting issue, do you observe any other abnormality ?
Well, yes, the node didn't seem to be actually at 100% memory use; as mentioned, it seemed to have significant headroom.
how about
Node Memory utilization is ratio of node Working Set Bytes and node allocatable memory. Allocatable memory is available on node object: Node Working Set Bytes are available on api: Please provide results from those commands so we can distinquish if problem is in kubectl top or in metrics pipeline.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have similar problem: kubectl describe node secondary-pool-dqiqzzikb5 kubectl get --raw I have no idea what's causing it or how to fix :/
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen We have the same issue.
Reopened this issue. <detailsIn response to : /reopen We have the same issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /triage accepted /assign
Seeing same issue here too
Another data point (on AWS EKS): Seems like happen only on smaller nodetypes (I see it with both micro/nano, others are medium) Usage is displayed correctly with NodeMetrics:
Why kubelet node memory utilization can exceed 100% (same can happen for CPU)? Simple, because it's not utilization of physical device, but utilization of resources allocated for pods and system daemons. How utilization of resources allocated is calculated? Simple, it's sum of resources used by on node divided by all resources allocated. How Kubelet allocates resources for pods? It takes all resources available on VM and substracts resources that it reserves for itself, kernel etc. How Kubelet knows how much resources it needs to reserve? User provides it in flags.

There is no error, there is no Kubernetes magic that gets us over 100%, it's just question of how you define utilization. Kubelet reserves some resources for system, they are not included when calculating node utilization, which means it can go above 100% when pods start using reserved resources (this is what over-committing means in "kubectl describe node" output). Some math based on your kubectl describe node output Node capacity: 968424Ki Node allocatable: 559848Ki Node Usage 717664Ki Node utilization: Node usage / node allocatable = 717664Ki / 559848Ki = 128% VM utilization: Node usage / node capacity = 717664Ki / 968424Ki = 74%
Based on my reading of the , I don't think "Node Usage" include only usage from pods. The includes usage by system daemons, so I'm not sure it makes sense to compare to allocatable, which is meant to exclude system daemons.
Hmm, I was not aware of that. This is a good point, as utilization was implemented in kubectl by different SIG (CLI), it's possible that it was not properly reviewed by other stakeholders (SIG Instrumentation/SIG Node). I would make sense to revisit what should be displayed as node utilization.


Question: func (f *FakeCache) FinishBinding(pod *v1.Pod) error { return nil } <del> func (f *FakeCache) ForgetPod(pod *v1.Pod) error { return nil } </del> <ins> func (f *FakeCache) ForgetPod(pod *v1.Pod) error { f.ForgetFunc(pod) return nil } </ins> func (f *FakeCache) AddPod(pod *v1.Pod) error { return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod got stuck in Pending state forever when failed to bind to host. What you expected to happen: For it to be rescheduled on another host How to reproduce it (as minimally and precisely as possible): Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs: Describe pod output: RC yml: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scheduling
Can you provide the details steps for this issue? And the output of ?
The log just showed the node was notReady or unreachable. Seems the kubelet node is not quite stable that the connections with the master always get lost.
Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide? no definite steps.

We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.
From the log, it seemed that failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes. Would you please append when you start and reproduce this issue. Also the kubelet log of that erroneous node.
^^ could you start the schedule with the above param next time for validation test run?
/cc
I ran our tests again, and hit this issue with several pods: There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with : :
Seeing this as well. hypothesis: broke the error handling when fails. Before that commit, when failed it would log an error and pass the pod to the error handler (), which is in charge of retrying scheduling the pod. After that commit, when fails it skips the error handling and scheduling is never retried. I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.
/cc
Can you post the description of per namespace?() Are you enabled ? And what's the config of component?
there are no specific parameters passed on the scheduler start besides address and cloudconfig
I posted a patch that i believe fixes this issue at -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. could you apply and see if your validation tests pass?
thanks!! We will be able to validate it either today or early next week
Are you able to validate the fix provided by through ? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it. cc/ the patch manager for 1.7 patch release.
our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week.

Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.
Thanks everyone who helped investigate this!
just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you
I have this problem with k8s v1.7.0 . How can I fix it, upgrading to v1.7.X? Thank you!
I am seeing this in 1.9. What steps I need to take to solve this problem?
I am seeing this periodically on Azure VMs
any chance you could open a new issue linked to this one with the requisite troubleshooting info (versions, etc.) and label it with ` Thanks!


Question: // ApplyFeatureGates applies algorithm by feature gates. func ApplyFeatureGates() { if utilfeature.DefaultFeatureGate.Enabled(features.TaintNodesByCondition) { <del> // Remove "CheckNodeCondition" predicate </del> <ins> // Remove "CheckNodeCondition", "CheckNodeMemoryPressure" and "CheckNodeDiskPressure" predicates </ins> factory.RemoveFitPredicate(predicates.CheckNodeConditionPred) <del> // Remove Key "CheckNodeCondition" From All Algorithm Provider </del> <ins> factory.RemoveFitPredicate(predicates.CheckNodeMemoryPressurePred) factory.RemoveFitPredicate(predicates.CheckNodeDiskPressurePred) // Remove key "CheckNodeCondition", "CheckNodeMemoryPressure" and "CheckNodeDiskPressure" // from ALL algorithm provider </ins> // The key will be removed from all providers which in algorithmProviderMap[] // if you just want remove specific provider, call func RemovePredicateKeyFromAlgoProvider() factory.RemovePredicateKeyFromAlgorithmProviderMap(predicates.CheckNodeConditionPred) <ins> factory.RemovePredicateKeyFromAlgorithmProviderMap(predicates.CheckNodeMemoryPressurePred) factory.RemovePredicateKeyFromAlgorithmProviderMap(predicates.CheckNodeDiskPressurePred) </ins> // Fit is determined based on whether a pod can tolerate all of the node's taints factory.RegisterMandatoryFitPredicate(predicates.PodToleratesNodeTaintsPred, predicates.PodToleratesNodeTaints)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug Description: If TaintNodesByCondition enabled, we should disable and predicates in scheduler. In TaintNodeByCondition, we will handle those conditions: , , and ; NetworkUnavailable checking is removed by predicates, but and not (no predicate for ). /cc
/sig scheduling /sig node
/assign


Question: If you are running the Vagrant based environment, the **salt-api** service is running on the kubernetes-master.  It is configured to enable the vagrant user to introspect the salt cluster in order to find out about machines in the Vagrant environment via a REST API. <ins> ## Standalone Salt Configuration on GCE On GCE, the master and nodes are all configured as [standalone minions](http://docs.saltstack.com/en/latest/topics/tutorials/standalone_minion.html). The configuration for each VM is derived from the VM's [instance metadata](https://cloud.google.com/compute/docs/metadata) and then stored in Salt grains (`/etc/salt/minion.d/grains.conf`) and pillars (`/srv/salt-overlay/pillar/cluster-params.sls`) that local Salt uses to enforce state. All remaining sections that refer to master/minion setups should be ignored for GCE. One fallout of the GCE setup is that the Salt mine doesn't exist - there is no sharing of configuration amongst nodes. </ins> ## Salt security <ins> *(Not applicable on default GCE setup.)* </ins> Security is not enabled on the salt-master, and the salt-master is configured to auto-accept incoming requests from minions.  It is not recommended to use this security configuration in production environments without deeper study.  (In some environments this isn't as bad as it might sound if the salt master port isn't externally accessible and you trust everyone on your network.) ```

We need to document all the configuration derived by Salt. Ideally this would be a markdown doc that we kept as live documentation within the project. The main reasons to bother: It's hard right now for anyone approaching the directory to get a big picture view of what's is going on, especially across providers. If we ever want to do something disciplined w.r.t. upgrade between (non-dev) versions, we're probably going to need a place to look for the state of the world. Decreasing the velocity with which people can add to Salt configuration will encourage people to delete more Salt configuration. cc


Question: import ( "crypto/sha1" <del> "encoding/base64" </del> <ins> "encoding/base32" </ins> "fmt" "io/ioutil" "os" "path/filepath" <ins> "regexp" </ins> "sort" <ins> "strings" </ins> "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet"

The hash function generates names with upper-case letters, but the validation only allows lower-case. First, we clearly need a test for this. Second, we need to fix it. Should we allow upper-case characters (which the RFC does allow for DNS labels) or should we fix the hash? Allowing upper-case seems attractive, except that the RFC also says that case doesn't matter, but should be preserved "wherever possible". So we really have three choices, I see. a) only allow lower-case b) allow upper-case and always use caseless comparisons everywhere c) allow upper-case, but normalize to lower-case since it was your hash code
Re: test - sounds like a config source -config integration test, which I'll add. Re: hash - would argue we should fix the hash to generate lowercase for now. I think, but will double check, that the collision space is still improbably low for 63 lowercase chars. I can do that shortly.
And a validate test on all sources.
Lowercase would require base32, rather than base64, no? On Aug 3, 2014 8:20 AM, "Clayton Coleman" wrote:
Yeah, although I don't believe there's a difference to the outcome between base32 and base64(lowercase) given an appropriate hash function.
Well, base32 is reversible and a modified base64is not :) On Aug 3, 2014 9:00 AM, "Clayton Coleman" wrote:
Remind me again why reversibility is important? We discussed it in another issue but I can't find the context.
I'm going to change the generated name to which even with 2^8 files per host and a thousand hosts should be well below 0.% chance of collision
Its only important if we want to use it. We don't AFAIK. On Aug 3, 2014 9:34 AM, "Clayton Coleman" wrote:
Fix makes them unique, somewhat traceable (first 15 safe chars of the name), and generates base32 output.


Question: // Indication the queue is closed. // Used to indicate a queue is closed so a control loop can exit when a queue is empty. // Currently, not used to gate any of CRED operations. <del> closed     bool closedLock sync.Mutex </del> <ins> closed bool </ins> } var (

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:


Question: subpather := subpath.New(mounter) hu := mount.NewHostUtil() var pluginRunner = exec.New() <del> if s.Containerized { klog.V(2).Info("Running kubelet in containerized mode") ne, err := nsenter.NewNsenter(nsenter.DefaultHostRootFsPath, exec.New()) if err != nil { return nil, err } mounter = nsutil.NewMounter(s.RootDirectory, ne) // NSenter only valid on Linux subpather = subpath.NewNSEnter(mounter, ne, s.RootDirectory) hu = nsutil.NewHostUtil(ne, s.RootDirectory) // an exec interface which can use nsenter for flex plugin calls pluginRunner, err = nsenter.NewNsenter(nsenter.DefaultHostRootFsPath, exec.New()) if err != nil { return nil, err } } </del> var dockerClientConfig *dockershim.ClientConfig if s.ContainerRuntime == kubetypes.DockerContainerRuntime {

There was a previous attempt : We have open issues that is not being worked on: I think it's time to delete the CI jobs and start the deprecation cycle. /sig node /assign
We discussed this at today's SIG Node meeting. SIG Node agreed to deprecate "containerized kubelet" with the following action items: The project was initiated for self-host cluster by SIG cluster lifecycle. We should communicate with them first to understand the potential dependencies. There were some demand from the community for this feature in the past, but we don't know who are using this in their production. We should announce this deprecation at Kubernetes Community weekly meeting and send email to kubernetes-dev
will do thanks
/sig storage /sig cluster-lifecycle
cc cc
Does this deprecate based in any way? I have a project that depends on this and I think it is a real use case for a container based . Running either a based or a dedicated container is key to our design.
it doesn't. the main thing to go would be this flag, if you are not using it, you are good. [1]
What is the reason behind this decision? We are heavily using containerized in production running on CoreOS. Would that mean that we no longer can run kubelet in a container?
Do you use the flag as shown here? Do you have mounts as shown just below the line pointed above If so, you have about a 3 releases per our deprecation policies to switch over to some other way of running kubelet. RedHat/CoreOS folks who initially championed this approach have given up on it, there are no test suites, no owners. sig-storage is having a hard time trying to keep things working. So please ping the CoreOS folks to research for an alternative for the future. Note that still runs kubelet in a container, it just does not use the flag or volume mounts as shown above.
Is it necessary to remove/deprecate the feature or just disable the flaky CI check? I find that feature and running kubelet in container helpful. Briefly looking at the implementation there is at least unit test suit.

that's the tip of the ice berg, there's a lot more things that can be simplified when we drop this feature, example see Another example of what we had to go through is here - There is no appetite to continue taking on this burden. I'll let and others chime in on the undue cost of keeping this alive.
fwiw I don't think we should deprecate the entire use case b/c it's highly valuable for testing. I'm not certain why this matters for the kubelet, b/c it really shouldn't unless it's a design flaw.
can you please chime in with the ongoing cost of supporting this feature? how much effort is to keep it going?
Yes we use this list of volume: This seems to work pretty nice for us and we don't have any volume issues so far.
Maintenance of containerized kubelet is increasingly painful. Kubelet / its storage code must know about all mounts on the host, resolve symlinks on the host, execute various helpers on the host and similar stuff, which is especially hard when it does not actually run on the host but in a weird container. See and for all the hacks we use. They are brittle and tend to break in very unexpected way. It's also largely undocumented and various ways how to run kubelet in a container are cargo-culted around. See above, , really? No offense, I know why it's there, but whole containerized kubelet is hack on top of hack on top of hack.
Not only is the code complex, but it is also not tested, so we have had to rely on manual testing when making bug fixes, and we don't know about regressions until after the release when users report issues. sig-storage has tried for 6+ months to find an owner that can dedicate resources to support and maintain this environment, but nobody has stepped up.
+1 to what asnd said. Either we should deprecate the feature, or find an owners who can commit to fixing issues and adding much needed tests. We tried to find an owner in SIG storage during Q2 2018 but no one stepped up.
Just because I like causing trouble, another alternative is to run executable from a container (or systemd portable unit, etc), but unify the kubelet's rootfs view with that of the host. Specifically: do the equivalent of , but for .

Then you would run from docker with (or equivalent), and early in chroot() to /rootfs. This loads the executable from the docker image, but all later filesystem operations are done from a pov equivalent to the host root fs. I think this keeps the kubelet lifecycle management benefits of running the kubelet binary from a container and allows the storage folks to just do regular filesystem operations, like there was nothing clever going on.
The function needs ownership from both SIG Node and SIG Storage. There were attempts early in the project to remove containerized kubelet support, but Red Hat offered contributors to maintain this model in both SIGs for ~11 releases of Kubernetes. Kubernetes was much smaller 11 releases ago and our shared project experience was limited. It's one thing for a feature to work initially, and another thing to reason about the impact of alternative deployment models when dealing with CVE responses or just doing normal maintainer review of incoming PRs or feature function. That lived experience is informing our future choices at Red Hat to focus on just deploying kubelet as systemd unit (like many other Kubernetes service providers), but if there are others in the community that want to keep the function alive and maintain it we need owners to volunteer and commit to fix issues and add much needed tests. If not, it seems the responsible thing for the community is to clearly deprecate it.
I like this suggestion to simplify the mechanism while retaining the packaging and runtime benefits of the container image. I'm curious what we're doing in talos if we're not using the flag and how what we're doing solves the use-case of running kubelet inside a container.
I kind of like the idea of mounting to as well. Although, one of the benefits we see in mounting only the needed host paths is that it is less of the host filesystem exposed in case of a security vulnerability. On the other hand we currently mount everything we think we need: and that could change in any release. The above solution would allow us to never worry about that again really. We don't use the flag, and I can't say I have seen any of the mentioned issues in our setup. We pass Sonobuoy conformance tests as well. So I can't say I would know exactly what we are doing to solve these problems since I personally have ran into them. I have had rook run just fine as well.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: CreateVolume(name, storageAccount, storageAccountType, location string, requestGB int) (string, string, int, error) // Delete a VHD blob DeleteVolume(diskURI string) error <ins> // Expand the disk to new size ResizeDisk(diskName string, oldSize resource.Quantity, newSize resource.Quantity) (resource.Quantity, error) </ins> } type azureDataDiskPlugin struct {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: According to , some volumes(e.g. aws, nfs, ceph) has already supported PV resizing, need to add PV resize support for azure disk What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: PV resize doc is here: Update: For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I will try to work on resizing an azure disk which is not in attached state. May dig into more details later. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Hi , is PV size grow feature still in alpha in master now? How do I open this feature, set in kubelet, kube-apiserer and kube-controller-manager config?
I have figured out that should be set in kubelet, kube-apiserer, kube-controller-manager, kube-scheduler config, and also need to enable in apiserver admission control. It works well.
should we go ahead and close this issue?
also file needs to be updated to allow resizing. This plugin provides additional checks for resizing.
For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I would suggest work on first, could you review , it works well in my testing env.
Adding priority important-soon to this issue so that it can stay in 1.10. If that is incorrect, please update. priority important-soon
/priority important-soon
PR has been merged, can this issue be closed?
yup, looks like it's in. Thanks for flagging! /close
you can't close an active issue unless you authored it or you are assigned to it, Can only assign issues to org members and/or repo collaborators.. <detailsIn response to : yup, looks like it's in. Thanks for flagging! /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Awww shucks, looks like I don't have magic powers. please close this when you have a chance?
has been merged, while for azure disk work is just in investigation state mainly because there is blocking issue: it does not support disk resize on a Running VM.
could someone move this issue out of milestone v1.10, thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Why was this closed? This is not fixed by that crappy PR. This is the kubernetes repo, and this refers to k8s apis being implemented correctly iyam. The PR that you closed this with does NOT support disk resize on a Running VM. Your words Please reopen this.
pls track this issue for online resize feature:


Question: // if we are going to issue an update, be sure we track every name we update so we know if we update it too often. currResult.updatedItems.Insert(fsu.flowSchema.Name) <ins> patchBytes, err := makeFlowSchemaConditionPatch(fsu.condition) if err != nil { // should never happen because these conditions are created here and well formed panic(fmt.Sprintf("Failed to json.Marshall(%#+v): %s", fsu.condition, err.Error())) } </ins> if klogV := klog.V(4); klogV.Enabled() { klogV.Infof("%s writing Condition %s to FlowSchema %s, which had ResourceVersion=%s, because its previous value was %s, diff: %s", cfgCtlr.name, fsu.condition, fsu.flowSchema.Name, fsu.flowSchema.ResourceVersion, fcfmt.Fmt(fsu.oldValue), cmp.Diff(fsu.oldValue, fsu.condition)) } fsIfc := cfgCtlr.flowcontrolClient.FlowSchemas() <del> applyOptions := metav1.ApplyOptions{FieldManager: cfgCtlr.asFieldManager, Force: true} // the condition field in fsStatusUpdate holds the new condition we want to update. // TODO: this will break when we have multiple conditions for a flowschema _, err := fsIfc.ApplyStatus(context.TODO(), toFlowSchemaApplyConfiguration(fsu), applyOptions) </del> <ins> patchOptions := metav1.PatchOptions{FieldManager: cfgCtlr.asFieldManager} _, err = fsIfc.Patch(context.TODO(), fsu.flowSchema.Name, apitypes.StrategicMergePatchType, patchBytes, patchOptions, "status") </ins> if err != nil { if apierrors.IsNotFound(err) { // This object has been deleted.  A notification is coming

A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted


Question: PREFIX = quay.io/fluentd_elasticsearch IMAGE = elasticsearch <del> TAG = v7.4.2 </del> <ins> TAG = v7.4.3 </ins> build: docker build --tag ${PREFIX}/${IMAGE}:${TAG} .

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: es-image as part of fluent-elasticsearch cluster addon does not build What you expected to happen: in the es-image folder should work How to reproduce it (as minimally and precisely as possible): cd cluster/addons/fluentd-elasticsearch/es-image make Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted
/sig instrumentation


Question: // DaemonSetUpdateStrategy defines a strategy to update a daemon set. type DaemonSetUpdateStrategy struct { // Type of daemon set update. Can be "RollingUpdate" or "OnDelete". <del> // Default is OnDelete. </del> // +optional Type DaemonSetUpdateStrategyType

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: the default DaemonSetUpdateStrategyType was changed to Ondelete by default, but the document still say that the default value was refer doc What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
refer the code for the default value of daemonset updateStrategytype
/sig docs
The default update strategy for Daemonset should be see here , I believe that code comment that you linked to is mistaken.
yep, I think the code comment is outofdate, I will fix it. /assign
/remove-kind bug /kind documentation


Question: "k8s.io/kubernetes/pkg/volume" ) <ins> const ( fileMode        = "file_mode" dirMode         = "dir_mode" vers            = "vers" defaultFileMode = "0700" defaultDirMode  = "0700" defaultVers     = "3.0" ) </ins> // Abstract interface to azure file operations. type azureUtil interface { GetAzureCredentials(host volume.VolumeHost, nameSpace, secretName string) (string, string, error)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): azurefile azurefile 0777 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Microsoft Azure Container Service OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Linux 4.4.0-28-generic -Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x8664 x8664 x8664 GNU/Linux What happened: I attempted to set up an Azure File Volume using the built-in volume type azureFile. Everything works, except that the files and directories defaults to permission 0777, always. When I tried to manually mount within the container, I could customise dirmode and filemode, meaning that this change should be possible. Reading the source code confirms it - dirmode=0777 and filemode=0777 seem hardcoded. What you expected to happen: Being able to supply dirmode and filemode to the mount. e.g. How to reproduce it (as minimally and precisely as possible): N/A. This is a feature request.

Anything else do we need to know: N/A
This is a big requirement, using an azure file volume for postgresql for instance fails due to permissions as initdb tries to change the permissions to be less restrictive but is unable to.
Evaluating compatibility between a certain setup of a volume mount and an arbitrary piece of software would be the responsibility of the integrator, wouldn't it? It seems a big requirement for K8s itself to have to ensure PostgreSQL can do what it wants. An integrator who tries to use Azure File Storage for PostgreSQL should know better to mount with the correct permission.
The point is that you cannot mount with permissions, they're hard coded to 0777.
Ah. Yeah. Sorry for misunderstanding the comment above. Let's say it is an "important" requirement :)
/sig azure
/assign andyzhangx
I have a for this, use mount options in storage class, below is an example:
Has this change made it into any release yet? I can't seem to find it... Or was the issue resolved in some other way? I'm facing the issue you originally reported, i.e. using a pod volume to an azureFile, and it always mounting with filemode and dirmode of 0755... and I need it to be 0777. Is there a workaround? TIA.
azure file mountOptions feature is available from v1.8.5, here is the example :
Thanks -- I was still having trouble until I the storageClassName: "" to the claim... then realized I was hitting issue which I can avoid by setting storageClassName: "default" to both the pv and pvc, and now everything works. Thanks again!


Question: trimmed = strings.TrimPrefix(newPath, "../") } <del> // trim leftover ".." if newPath == ".." { </del> <ins> // trim leftover {".", ".."} if newPath == "." || newPath == ".." { </ins> newPath = "" }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I asked kubectl to extract all the files in a container. Only the names of the top level entries are concerned. How to reproduce it (as minimally and precisely as possible): Just cp the root folder somewhere. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-27T17:05:32Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: MaaS + juju OS (e.g. from /etc/os-release): Ubuntu 16.04.5 Kernel (e.g. ): 4.4.0-137-generic -Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: juju bundle from canonical
/assign
Hi, - sorry, I didn't notice that this was assigned to you. Would you like me to close my PR?
/sig cli
never mind


Question: }, }, options: printers.GenerateOptions{}, <del> // Columns: Name, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job3", "0/1", "30m", "10y"}}}, </del> <ins> // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job3", "Running", "0/1", "30m", "10y"}}}, </ins> }, { job: batch.Job{

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: if f.Timeout != nil { options.Timeout = *f.Timeout } <ins> if f.Wait != nil { options.WaitForDeletion = *f.Wait } </ins> return options }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc