Question: defer cancel() c, err := grpc.DialContext(ctx, unixSocketPath, <ins> grpc.WithAuthority("localhost"), </ins> grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock(), grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {

When registering a new driver the client is sending the filepath of the socket as the / header instead of . In practice almost all / servers outright reject this as invalid. Given that the client was updated years ago I'm unclear why this problem still persists to this day :( kubelet logs look like this: csi server logs look like this: Registration to move along happily. simple instructions here: Most drivers are written in and the server simply ignores this non-compliant header, otherwise the issue would be far more prevalent. <detailsEvery version up until current AFAIK. </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
I ran into this error in my use-case and debugged what's going on. I think the problem goes away with a recent enough and when the socket path is correctly prefixed in the client connection: I though I could start working on fixing this in kubelet.
That would be awesome! Then I can finally do away with all the crazy workarounds.
The nomad issue link you gave tried the approach I was talking about without luck it seems. I need take a closer look what's wrong...
btw, this look a duplicate of and
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale currently planning to use but I have some cleanup pending with first
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale The fix PR is pending approval

When using the CSI driver node registration sidecar container, the kubelet-registration-path parameter is set to either or . One or both of these options should cause kubelet to send a valid authority header to the socket. In the former case, kubelet will fail to find the file path since it will pass the unix header in the net.Dialer target to dial. In the latter, the dialer will succeed to call the container over the socket but send an incorrect :authority pseudo header (the ). A call into the CSI container with a valid authority header, using one or either kubelet-registration-path parameter. Deploy a CSI plugin example, setting kubelet-registration-path on the node-driver-registrar sidecar container to either a or . Note, a plugin may not fail in the latter case if the CSI plugin is written in a language with an http2 library that does not strictly check the authority header, however the header will still be incorrect. If using Rust as a language, the h2 library will strictly check the authority header and return a protocol error. Go does not seem to reject the invalid authority header, which is perhaps why most plugins do not notice the issue. Using will not have a 'unix:' header. It checks for this to substitute 'localhost' as the authority , which will not happen in this case. Using will get the authority substituted, but will pass the full 'unix:' header in as part of the path file. Related code can be seen (non-nil custom dialer) and (newGrpcConn looks like it expects no unix header based on log entry and externally supplied dialcontext). This work may have been overlapping with related work in grpc-go and plans , where both libraries seem to be taking responsibility for managing the authority header for unix sockets now. Kubelet logs using the unix header: Of note: the inconsistent 'Error while dialing dial unix' entries: one showing the unix: header for the path, while the registration socket shows it without. Also, the 'localhost' replacement is visible in the logs for the first (unix:-prefixed) and not the second registration notification call (non-prefixed). node registration sidecar logs when not_ using the unix header: Of note, kubelet notifies the container it received a protocol error.

The CSI container rust logs show the matching protocol error and authority header value: <details</details<detailsBare metal </details<details</details<details</details<detailscri-o </details<detailscalico, metal-lb, bgp, applicable 1.23 versions. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

ref: Target should start with unix: to avoid setting empty authority header according to RFC-3986.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
looks same as and
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue.


Question: }, }, { <del> ObjectMeta: metav1.ObjectMeta{Name: "system:aws-cloud-provider"}, Rules: []rbacv1.PolicyRule{ rbacv1helpers.NewRule("get", "patch").Groups(legacyGroup).Resources("nodes").RuleOrDie(), eventsRule(), }, }, { </del> // a role making the csrapprover controller approve a node client CSR ObjectMeta: metav1.ObjectMeta{Name: "system:certificates.k8s.io:certificatesigningrequests:nodeclient"}, Rules: []rbacv1.PolicyRule{

As of , all Kubernetes clusters with RBAC bootstrap roles for AWS, regardless of whether they are running on AWS. The hardcoded RBAC bootstrap policy should only include roles that are used by all clusters, regardless of addons, configuration, or cloud provider. The ClusterRole(Binding)s should be removed from the bootstrap policy, and instead configured by the provisioning tools. /sig aws /sig cloud-provider /sig auth
/assign I can help do it. :)


Question: echo "VAGRANT_DEFAULT_PROVIDER='${VAGRANT_DEFAULT_PROVIDER:-}'" echo "KUBELET_TOKEN='${KUBELET_TOKEN:-}'" echo "KUBE_PROXY_TOKEN='${KUBE_PROXY_TOKEN:-}'" <ins> echo "MASTER_EXTRA_SANS='${MASTER_EXTRA_SANS:-}'" </ins> awk '!/^#/' "${KUBE_ROOT}/cluster/vagrant/provision-network.sh" awk '!/^#/' "${KUBE_ROOT}/cluster/vagrant/provision-master.sh" ) > "${KUBE_TEMP}/master-start.sh"

On latest Kubernetes version Release 0.20.1 Candidate I get the following when you setup cluster with "./cluster/kube-". This is on Mac OSX with VirtualBox, Vagrant Any ideas, thanks .... ==minion-3:  INFO: Running daemons_running() ==minion-3:  INFO: Salt installed! Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Validating minion-1 Validating minion-2 Validating minion-3 ..... Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused kubernetes user1$ get nodes error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
the same for the first time when i ran v0.20.1 (the published one)
here is the installation log on my core i5-5200U ubuntu 14.04 3.16 kernel this is that console says
See my comment on
I think is probably right. There's a 0.20.2 brewing already (and we should have it out later today), so hang tight.
I just tried this also with Release 0.20.2 Candidate on Ubuntu 14.04 with VirtualBox, Vagrant and I am seeing:
- Is there someone on your team that could take a look?
Honestly no, everyone is busy, and working on either cluster or network related issue. But I talked to briefly, he kindly agree to take a look. The cloud provider here is Vagrant, if you could reproduce the issue, that would be huge help. Feel free to re-assign it to me if you are blocked. Thanks!
cc/ since he looked into vagrant issue lately. cc/ on Vagrant.

+1, 0.20.2 still has this issue
+1
The fix for this issue is this PR:
Assigning issue to me since I am focused today on closing out remaining Vagrant setup issues.
Sigh, I'm sorry. So busy working through unassigned PRs I forgot to work through my own queue.
I was running into the same issue and cherry picked your changes in , but still have the same issue. Prior to those changes the kubelet never started on the master. With the changes it does start, but I still get: It looks like also needed before vagrant works correctly.
Yep, I picked and then merged and it started to work again... just my 2 cents
this error is gone in 0.21.0 rc ()
I am using Mac OSX 10.10.5 / VMWare Fusion / bento/fedora-21/ Vagrant 1.7.4 / and built kubernetes from master (GitVersion:"v1.1.0-alpha.1.687+") and I am seeing this issue when I run ./cluster/kube- Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Connection to 192.168.100.177 closed. ..Validating minion-1 ............. Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
I'm running into a similar issue with a Kubernetes cluster on AWS. I set it up last week and now can't connect to the master node: oamasood kubectl cluster-info error: couldn't read version from server: Get https://11.11.111.111/api: dial tcp 11.11.111.111:443: connection refused Any help would be appreciated.
I find that if you wait for the master and minion to calm down on activity after a bit (~5 minutes) the kubectl commands work, however the problem is I don't see any nodes running.

I have re-installed virtual box, made sure my .kube directory is clean and I am now pulling just the stock vagrant box cube fedora 21 and kubernetes 1.0.4 release. I do not see any docker processes running on my minion. Other than that it seems that the cluster is running: $ get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok nil scheduler Healthy ok nil etcd-0 Healthy {"health": "true"} nil $ cluster-info Kubernetes master is running at https://10.245.1.2 KubeDNS is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-dns KubeUI is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-ui yggdrasil:kubernetes jonathantronson$ vagrant ssh master Last login: Wed Sep 16 23:49:51 2015 from 10.0.2.2 [vagrant ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES "/bin/sh -c '/usr/loc" 50 minutes ago Up 50 minutes k8skube-apiserver.2abef58bkube-apiserver-kubernetes-masterkube-system496e220b4704d4b390883b4b90aa66354e41e61f "/bin/sh -c '/usr/loc" 50 minutes ago Up 50 minutes k8setcd-container.fa2ab1d9etcd-server-kubernetes-masterkube-system6ef1b07678064a7edcdbec6f0570e91759d753f9 "/bin/sh -c '/usr/loc" 53 minutes ago Up 53 minutes k8skube-scheduler.ea78c200kube-scheduler-kubernetes-masterkube-system3d1ced0542467b566546021cb74d9bdf9550be6a "/bin/sh -c '/usr/loc" 53 minutes ago Up 53 minutes k8skube-controller-manager.fd0f0b82kube-controller-manager-kubernetes-masterkube-system2f59d05ecb2212d0379dc35dbe40d5216292ac06 "/pause" 54 minutes ago Up 54 minutes k8sPOD.

e4cc795kube-scheduler-kubernetes-masterkube-system3d1ced0542467b566546021cb74d9bdf62e6c529 "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795kube-controller-manager-kubernetes-masterkube-system2f59d05ecb2212d0379dc35dbe40d52109062f04 "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795etcd-server-kubernetes-masterkube-system6ef1b07678064a7edcdbec6f0570e9173d73246b "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795kube-apiserver-kubernetes-masterkube-system496e220b4704d4b390883b4b90aa6635c1bfe2c3 [vagrant ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
The only errors in my minion salt log are: 2015-09-16 23:40:22,049 [][ERROR ][2668] Command 'systemctl status docker.service' failed with return code: 3 2015-09-16 23:40:22,049 [][ERROR ][2668] output: * docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled) Active: inactive (dead) Docs:
Hi, I am running the latest version of kubernetes tar ball (v1.1.3) and facing issues with ./kube- I was unable to fix the issue. Any help would be appreciated. cluster "vagrant" set. user "vagrant" set. context "vagrant" set. switched to context "vagrant". Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Connection to 127.0.0.1 closed.

Validating minion-1 Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused Thank You
Mac 10.11.2 installing kubernetes 1.1.3 got same issue: I think the fix didn't catch the root cause.
same issues on 1.1.7 cluster "vagrant" set. user "vagrant" set. context "vagrant" set. switched to context "vagrant". Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master ^ to 127.0.0.1 closed. Validating minion-1 ...^ Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
when I rerun the cluster/kube-, the error still exist, and can't get nodes/services etc status. The first time, it works. weird.
can u help to take a look this issues? thanks
- can you open a new issue and describe what host environment you are using? Are you testing from master or a branch? cc me on the issue.
I used released version 1.1.7. btw, I tried several times. even at the last step failed. I can run get pods/services etcs. seems during verify stage. it will search local kubectl files? I am not sure. as on my Mac , there have some mismatch version kubectl file.
I experience the very same issue on AWS and, usually, I can't connect to my cluster anymore the day after I had set it up - which is a huge blocker. Here's the output from :
I expreience the same issue on Ubuntu.
The issue disappeared for me with the following version:
I encounter the same problem,can you give me a detail configuration of kuberenetes?

ubuntu /opt/bin/kubectl version Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.8", GitCommit:"", GitTreeState:"clean"} error: couldn't read version from server: Get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused


Question: err      error after    int <del> parallelLock sync.Mutex parallel     int maxParallel  int delay time.Duration </del> <ins> // this block should be updated consistently parallelLock                sync.Mutex shouldTrackParallelRequests bool parallelRequests            int maxParallelRequests         int parallelRequestDelay        time.Duration </ins> } <del> func (rt *requestTracker) errorReady() bool { rt.Lock() defer rt.Unlock() return rt.err != nil && rt.requests >= rt.after } func (rt *requestTracker) inc() { rt.parallelLock.Lock() rt.parallel++ if rt.maxParallel < rt.parallel { rt.maxParallel = rt.parallel </del> <ins> func (rt *requestTracker) trackParallelRequests() { if !rt.shouldTrackParallelRequests { // do not track parallel requests unless specifically enabled return </ins> } <del> rt.parallelLock.Unlock() rt.Lock() defer rt.Unlock() rt.requests++ if rt.delay != 0 { time.Sleep(rt.delay) </del> <ins> if rt.parallelLock.TryLock() { // lock acquired: we are the only or the first concurrent request // initialize the next set of parallel requests rt.parallelRequests = 1 } else { // lock is held by other requests // now wait for the lock to increase the parallelRequests rt.parallelLock.Lock() rt.parallelRequests++ } defer rt.parallelLock.Unlock() // update the local maximum of parallel collisions if rt.maxParallelRequests < rt.parallelRequests { rt.maxParallelRequests = rt.parallelRequests } // increase the chance of collisions if rt.parallelRequestDelay > 0 { time.Sleep(rt.parallelRequestDelay) </ins> } } <del> func (rt *requestTracker) reset() { rt.parallelLock.Lock() rt.parallel = 0 rt.parallelLock.Unlock() rt.Lock() defer rt.Unlock() rt.err = nil rt.after = 0 rt.delay = 0 } func (rt *requestTracker) getErr() error { </del> <ins> func (rt *requestTracker) incWithOptionalError() error { </ins> rt.Lock() defer rt.Unlock() <del> return rt.err </del> <ins> rt.requests++ if rt.err != nil && rt.requests >= rt.after { // reset and pass the error defer func() { rt.err = nil rt.after = 0 }() return rt.err } return nil </ins> } func newRequestTracker(requests int, err error, after int) requestTracker {

https://prow.ppc64le- TestStatefulSetControl/CreatePodFailure/Burst/ScaleDownOnly/StatefulSetAutoDeletePVCEnabled of Have started observing more flakes recently since last week. No response_ Able to reproduce using stress tool. /sig testing /sig apps
/triage accepted /assign
opened a PR with a fix:


Question: return nil } <del> b := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand(cmd)). </del> <ins> b := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand()). </ins> NamespaceParam(cmdNamespace).DefaultNamespace(). SelectorParam(selector). ResourceTypeOrNameArgs(true, args...).

I think the need for this argument to the factory methods has passed - it means that callers who want to provide configurable behavior have to do it through the factory itself, which is acceptable. We can remove it whenever.
I removed most of the unused arguments. But, There are a few methods still using the argument. Namely, , , and . Before proceeding any further I wanted to ask what should be done about these methods? Deprecated/Refactored/Removed ?
The factory methods were the most important. The pattern before was people using the flags by name (cmd.Flags().Get("foo")), but now factory methods must use pointer flag vars, so there's nothing that should need the command there. The printer methods are optional units that can be used when you add the flags for them to your command. Unless we refactor the definition of those flags to use pointers and return a printer struct (something I had started a while back and never finished), it's not worth changing them right now. The other consumer of factory is OpenShift - we wrap the factory to add our additional types (implementations of things like describers, action verbs, etc): - but we will pick up and remove use of cmd when we rebase.
Thanks for the explanation. I've cleaned the as well. Print method APIs are untouched as you suggested.
cc
Looks like this was done by PR


Question: // ComponentPod returns a Pod object from the container, volume and annotations specifications func ComponentPod(container v1.Container, volumes map[string]v1.Volume, annotations map[string]string) v1.Pod { <ins> // priority value for system-node-critical class priority := int32(2000001000) </ins> return v1.Pod{ TypeMeta: metav1.TypeMeta{ APIVersion: "v1",

etcd / kube-apiserver stops before most containers, despite using . On a single node server, with multus (and likely other CNI talking to the API), this spam the logs and make the shutdown slower should stop one group at a time On a single node cluster, have a slow to shutdown pod (sleep inf), look at the logs to see in which order they are killed Reading the kep , the summary says so I would expect all pods with priority between 0 and to be stopped first, then coredns (), then the rest. This is definitely not what's happening: <details</detailsN/A <details</detailskubeadm containerd multus
/cc /sig node
Each will match all Pods whose is less than or equal to it. This may suit your want
/kind support
sorry for the delay I missed the notification ... this is the opposite of what the documentation says :( Maybe i'm misunderstanding the feature, I want shutdown ordering, and it's not working for me. All my pods have priority set to either 0, or . I want all the pods with priority 0 to shutdown first, once they are all shutdown or grace period is elapsed, shutdown the ones with priority , wait for them to shutdown or grace period, and finally take care of the ones with priority . Here some failing tests: test1: First killed pod was etcd (verified priority is ) ... test2: first pod killed was kube-scheduler test3 (same as test2): first pod killed was kube-controller-manager test4: first pod killed was etcd test5 (config in reverse order as test1): etcd was killed 3rd but still before pods with priority 0
I see what you mean, but this shouldn't be happening. It should only shutdown based on the priority of the group from lowest to highest. I would like to know the priority of all Pods on the node you shut down, and the configuration of , to try to reproduce it on my side
I'm reproducing it on an empty cluster, but just noticed coredns is killed correctly, it's just all containers from last group end up in first group
For some reasons the system-node-critical / pods go with the priority 0 pods
I think the pods quit too quickly so it looks like they were deleted together, maybe you should block them until the last moment.

I think I found a lead, it's all the static pods that are killed too soon, not kubeproxy () they have priorityClassName set, but not priority, so maybe priority is nil when getting of a static pod
I see that in numeric priorities to manifests in cluster/gce/manifests/ with the comment in the PR , are priority mandatory for static pods ? should we make kubeadm add them, or fix that in kube somewhere ?
This is the result from the API, I bet kubelet has different path to get pod manifests for cluster pods and static pods, as kubelet can work without the API
If I manually add to etcd manifest it's killed in the last group with kubeproxy
/remove-kind support
/triage accepted
[For a single-node cluster], I wonder if we should actually set a higher priority than for all or a subset of control plane Pods, because even with this change we could still shut down the API server quickly and before a node-critical component has completed its own shutdown.
I agree with this idea, because of the dependencies between the components of node-critical.


Question: expectedReplicas:             255, // (100 - 15) + 200% expectedCondition:            "ScaleUpLimit", }, <ins> { name:                         "scaleUp with percent policy and previous scale up and down events", scaleUpEvents:                generateEventsUniformDistribution([]int{4}, 120), scaleDownEvents:              generateEventsUniformDistribution([]int{2}, 120), specMinReplicas:              1, specMaxReplicas:              1000, scaleUpRules:                 generateScalingRules(0, 0, 300, 300, 0), currentReplicas:              6, prenormalizedDesiredReplicas: 24, expectedReplicas:             16, expectedCondition:            "ScaleUpLimit", }, </ins> // ScaleDown with PeriodSeconds usage { name:                         "scaleDown with default policy and previous events",

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: var _ volume.DeviceMounter = &csiAttacher{} func (c *csiAttacher) Attach(spec *volume.Spec, nodeName types.NodeName) (string, error) { <ins> _, ok := c.plugin.host.(volume.KubeletVolumeHost) if ok { return "", errors.New("attaching volumes from the kubelet is not supported") } </ins> if spec == nil { klog.Error(log("attacher.Attach missing volume.Spec")) return "", errors.New("missing spec")

The Kubelet panics when: 's started with the option; a CSI volume is attached to a Node. From the Kubelet logs: Either document that this is not supported (and stop the panic'ing) or support and test this use case. kubelet with the option set. a CSI PVC and use it in a pod. No response <details</details<detailsany </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/assign


Question: initializing := false for i := range pod.Status.InitContainerStatuses { container := pod.Status.InitContainerStatuses[i] <ins> restarts += int(container.RestartCount) </ins> switch { case container.State.Terminated != nil && container.State.Terminated.ExitCode == 0: continue

The output looks like below: What is expected behavior? Does restart counter should increase when init containers are failing? Does restart counter should be reseted when InitContainer will start work and normal containers start? cc
Restarts are tracked per container - I'd be concerned that the restarts the user is concerned with don't map exactly to the init container restarts, although knowing how many attempts to init have been made are useful. Since init containers are processed serially, when an init container passes, do we reset the restart count to zero? I think we could show the current init container restart count (if pod is in init) in this field. But once the init container succeeds it has to drop back to zero.
For normal containers field shows the sum of from containers status. I think for init containers as we know which container is running (when is merged) we can show restart count for current init container. For normal containers behavior should not change. So it would look like these: First init container is failing Second init container is failing Normal container is failing For better readability we may also add word like in status: Second init container is failing What do you think?
Yeah, although I don't know that we need the Init prefix in the status. I think users can reasonably be expected to infer that RESTARTS is subordinate to STATUS. The init prefix makes it harder to read without adding a ton. , Łukasz Oleś wrote:
Ah, I messed up. shows status of normal containers. in Status shows status of init containers: I think this part should be always showed. So it will look like these: What do you think?
Ok, I decided to use the same approach as for normal containers. RESTARTS just shows sum of all init containers restarts.
Fix is merged


Question: flags.StringVar(&TestContext.KubeContext, clientcmd.FlagContext, "", "kubeconfig context to use/override. If unset, will use value from 'current-context'") flags.StringVar(&TestContext.KubeAPIContentType, "kube-api-content-type", "application/vnd.kubernetes.protobuf", "ContentType used to communicate with apiserver") <del> flags.StringVar(&TestContext.KubeVolumeDir, "volume-dir", "/var/lib/kubelet", "Path to the directory containing the kubelet volumes.") </del> <ins> flags.StringVar(&TestContext.KubeletRootDir, "kubelet-root-dir", "/var/lib/kubelet", "The data directory of kubelet. Some tests (for example, CSI storage tests) deploy DaemonSets which need to know this value and cannot query it. Such tests only work in clusters where the data directory is the same on all nodes.") flags.StringVar(&TestContext.KubeletRootDir, "volume-dir", "/var/lib/kubelet", "An alias for --kubelet-root-dir, kept for backwards compatibility.") </ins> flags.StringVar(&TestContext.CertDir, "cert-dir", "", "Path to the directory containing the certs. Default is empty, which doesn't use certs.") flags.StringVar(&TestContext.RepoRoot, "repo-root", "../../", "Root directory of kubernetes repository, for finding test files.") // NOTE: Node E2E tests have this flag defined as well, but true by default.

What happened: The CSI hostpath driver used in the e2e tests expects the root directory of nodes to be set to , , however nodes can be configured to use different paths. The tests should find out the root directory from kubelet to allow for this. Several test failures related to CSI Volumes: What you expected to happen: Test should be expected to pass. How to reproduce it (as minimally and precisely as possible): Configure kubelet on cluster nodes to use the non-default path. Run the storage-csi e2e tests. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/sig storage /remove-sig testing This is a SIG storage specific issue.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
How? I've asked about that before and the conclusion was that there is no portable way to do that (i.e. no official API). It seems to fall onto cluster admins to configure CSI driver deployments so that this particular aspect matches the cluster. For the tests that means that there has to be a parameter for the kubelet data directory that users of then have to set.
We could add and respect a flag of course then all nodes have to be the same at least. Reminder though
That's indeed another complication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale Tracked for csi-driver-host-path in There's a bigger issue about both mock and hostpath driver in which includes "support non-default kubelet path".
The current hostpath driver in k/k testing supports non-default paths, but the suite doesn't use that yet.

/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: return nil, fmt.Errorf("Source range restrictions are not supported for openstack load balancers without managing security groups") } <del> affinity := v1.ServiceAffinityNone </del> <ins> affinity := apiService.Spec.SessionAffinity </ins> var persistence *v2pools.SessionPersistence switch affinity { case v1.ServiceAffinityNone:

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Creating service(type: LoadBalancer, sessionAffinity: ClientIP), service has no affinity. What you expected to happen: The service has affinity How to reproduce it (as minimally and precisely as possible): a service, its yaml as following: kind: Service apiVersion: v1 metadata: labels: app: application name: application spec: type: LoadBalancer ports: port: 80 targetPort: 9091 sessionAffinity: ClientIP selector: app: application request from Neutron, check the value of "Persistence" Anything else we need to know?: Kubernetes's OpenStack cloudprovider can't set persistence to "SOURCE_IP". Environment: Kubernetes version (use ): kubernetes 1.6.4
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig network
/assign
Fixed in /close


Question: for _, ipt := range ipts { encounteredError = userspace.CleanupLeftovers(ipt) || encounteredError encounteredError = iptables.CleanupLeftovers(ipt) || encounteredError <del> encounteredError = ipvs.CleanupLeftovers(s.IpvsInterface, ipt, s.IpsetInterface, s.CleanupIPVS) || encounteredError </del> <ins> encounteredError = ipvs.CleanupLeftovers(s.IpvsInterface, ipt, s.IpsetInterface) || encounteredError </ins> } if encounteredError { return errors.New("encountered an error while tearing down rules")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy don’t clean ipvs rules with . What you expected to happen: kube-proxy clean ipvs rules with by default. How to reproduce it (as minimally and precisely as possible): Run kube-proxy in ipvs mode, and clean rules with . Anything else we need to know?: Based on , should cleanup all rules by default, including ipvs rules. And also retains functionality. Environment: Kubernetes version (use ): v1.20.0-bate.0 Cloud provider or hardware configuration: OS (e.g: ): centos7 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
it seems that this flag entered deprecation, so I think that recovering functionaly is not the right approach you were involved in the discussion with this flag in , can you take a look at this?
thanks for your reply. The was in , to avoid removing IPVS rules which are not created by kube-proxy. According to , It seems that deleting ipvs rules by default is expected, but it is not so. I think that keeping --cleanup-ipvs is meaningful, unless kube-proxy can only delete the IPVS rules created by itself.
/assign
sorr for the late answer , but the flag must have the same functionality than , actually I can see that it has implemented. In case that the is not working we should fix it, but not recover the deprecated flag
Agree
sorry if I'm missing something. I just try to make the function of as described, not to recover the flag. The problem I see is: The can't flush IPVS. Because the IPVS flush process is still controlled by . Beforce , it was assigned by flag, but now it is not assigned, always false. currently does not have any functionality, just stored by . So, should we keep the function of temporarily by assigning it to ,or just make always flushing IPVS regardless of the ?

I prefer the former because the flag is not completely abandoned :)
it seems that is what the flag description says, you have to use to flush the ipvs entries if that is the case it seems that we are ok, right? no changes needed
In fact, can't flush ipvs entries. The reason is what I described above.
I´m sorry, I think I got it know, the flag is not exposed, but the option is there and initialized to true, this is what I see so, it seems that flush IPVS by default
yes, should flush IPVS by default, but it doesn’t actually
... and finally I got it, so sorry :bow: the ProxyServer is not being instantiated with the flag, despite is set :facepalm: you are more familiar with the IPVS implementation than me, I personally think that, if the flag deprecation has started, we should make that always flush IPVS and totally remove ... it seems aligned with the other proxy implementations
/triage accepted
Actually, after doing some code archaeology, this was the expectation per And the flag clearly mentions that --cleanup must flush IPVS entries too I think that we can go ahead and remove the --cleanup-ipvs flag and make --cleanup to flush ipvs too, right now is a mess and is not working at all , are you going to submit a PR?
thanks for digging up the history on this. That lines up with my recollection and the KEP around kube-proxy's clean up logic
OK, I will create a PR to remove the flag


Question: f *framework.Framework, sa *v1.ServiceAccount, ) *v1.Pod { <del> podClient := client.CoreV1().Pods(config.Namespace) </del> priv := true mountPropagation := v1.MountPropagationBidirectional hostPathType := v1.HostPathDirectoryOrCreate

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The code creates a pod directly, without waiting for the service account secret to be available. Depending on timing, this can fail with a error. What you expected to happen: It should use a DaemonSet instead, then the daemon set controller will ensure that the pod gets created eventually. [Edit: when implementing the fix, I found it easier and less intrusive to just retry pod creation.] How to reproduce it (as minimally and precisely as possible): It's not possible to trigger the failing race condition reliably. I've seen it a few times on my own cluster. Anything else we need to know?: Originally discussed in and in the CSI working group meeting on Monday, September 17th ().
/sig storage


Question: "net/http" "net/http/httptest" "reflect" <ins> "regexp" </ins> "strings" <ins> "sync" </ins> "testing" <ins> "time" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" )

Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)


Question: }, }, { <del> name: "spec:v4 ip:v4 dualstack:on", </del> <ins> name: "spec:v4 ip:v4 dualstack:on isIPv6Primary:false", isIPv6Primary:          false, </ins> specExpctPrimary:       true, clusterIPExpectPrimary: true, enableDualStack:        true,

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: From v1.24 onwards, you will need to either use one of the other [supported runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes/) (such as containerd or CRI-O) or use cri-dockerd if you are relying on Docker Engine as your container runtime. For more information about ensuring your cluster is ready for this removal, please <del> see [this guide](/blog/2022/03/31/ready-for-dockershim-removal/). </del> <ins> see [this guide](https://kubernetes.io/blog/2022/03/31/ready-for-dockershim-removal/). </ins> ### Beta APIs Off by Default

has some broken links. These are relative links that work OK in the release blog article but need to be absolute hyperlinks in the release notes. Links in release notes work. Visit Click on all the links. Specifically, “this guide” in (there are some more, this is one example) No response v1.24.0 n/a n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release and IMO /priority important-soon
Hi Tim, I can help fix this /assign


Question: #   --boilerplate <string = path_to_kube_codegen_boilerplate> #     An optional override for the header file to insert into generated files. # <ins> #   --extra-peer-dir <string> #     An optional list (this flag may be specified multiple times) of "extra" #     directories to consider during conversion generation. # </ins> function kube::codegen::gen_helpers() { local in_pkg_root="" local out_base="" # gengo needs the output dir must be $out_base/$out_pkg_root local boilerplate="${KUBE_CODEGEN_ROOT}/hack/boilerplate.go.txt" local v="${KUBE_VERBOSE:-0}" <ins> local extra_peers=() </ins> while [ "$#" -gt 0 ]; do case "$1" in

Earlier it was possible to pass flag to while calling script. Now with the new script, is called under which doesnot support any other flags apart from , and . As it can be seen here there is no option to pass flag to conversion-gen Earlier in the script, it was considering all the inputs passed flag is supported by under the hood, but the function in script is not allowing the flag.
/cc
/sig api-machinery
/assign for code generation /triage accepted
/assign
Can you explain why you need this flag? What I found in k/k was that we don't actually need it at all.
You know what, my memory was faulty. We do use it (until I fix that, which is in a development branch, not merged). But we don't use it in any of the "submodule" configs.
A lot of the "magic" behind the conversion logic is very brittle. Can you help me understand where / how this impacts you?
I whipped up a quick patch but have not tested much:
We are using the flag several places in our script On removal of the flag, the autoConvert functions in. files has the error : Althoug I can see the function is already present at . But there is also a comment like Hence, I think there is a need for the in order to use this pkg.
I see. The component-base is something I didn't account for. If you apply the above patch, does the new argument to work for you?
Above patch throws an error for the cases when flag is not passed to function. Instead I followed this approach :
I'd rather not accumulate into a string - can you tell me what error it throws? Note, mine wants Also - is the argument here a dir (e.g. /home/me/src/foo/bar) or a pkg (e.g. ) ? I think it's actually a pkg, right?
You implementation throws error for the cases when we don't specify the option while calling , as shown below: For such cases I am getting following error
It seems to be an old-bash problem. Try this one?
Yes this is working fine.
OK. I'll prep a PR soon.
Thank you


Question: continue } if secret.Type == v1.SecretTypeServiceAccountToken { <del> tokenContent = string(secret.Data[v1.ServiceAccountTokenKey]) </del> rootCAContent = string(secret.Data[v1.ServiceAccountRootCAKey]) return true, nil }

https://k8s- /sig auth /kind failing-test
Adding a priority to this: /priority important-soon Please raise the priority if you consider this a release blocker. Given the test job, it normally wouldn't be.
It's a test issue. I'm hoping to fix this before 1.13 but I don't consider it a release blocker.


Question: } oldNodeUnschedulable = node.Spec.Unschedulable } <ins> // NOTE(aaronlevy): NodeReady condition needs to be the last in the list of node conditions. // This is due to an issue with version skewed kubelet and master components. // ref: https://github.com/kubernetes/kubernetes/issues/16961 // TODO: Remove the code after we drop support for v1.0 kubelets lastIndex := len(node.Status.Conditions) - 1 for i := range node.Status.Conditions { if node.Status.Conditions[i].Type == api.NodeReady && i < lastIndex { node.Status.Conditions[i], node.Status.Conditions[lastIndex] = node.Status.Conditions[lastIndex], node.Status.Conditions[i] break } } </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was looking into what tests were enabled for and found that the following tests: are running 1.18. From the output of the tests (): and What you expected to happen: tests to be running against the latest code for 1.19 How to reproduce it (as minimally and precisely as possible): It looks to be caused by the setting: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig release /sig windows


Question: }) }) <del> func execSourceipTest(pausePod v1.Pod, serviceAddress string) (string, string) { var err error var stdout string timeout := 2 * time.Minute framework.Logf("Waiting up to %v to get response from %s", timeout, serviceAddress) cmd := fmt.Sprintf(`curl -q -s --connect-timeout 30 %s/clientip`, serviceAddress) for start := time.Now(); time.Since(start) < timeout; time.Sleep(2 * time.Second) { stdout, err = framework.RunHostCmd(pausePod.Namespace, pausePod.Name, cmd) if err != nil { framework.Logf("got err: %v, retry until timeout", err) continue } // Need to check output because it might omit in case of error. if strings.TrimSpace(stdout) == "" { framework.Logf("got empty stdout, retry until timeout") continue } break } framework.ExpectNoError(err) // The stdout return from RunHostCmd is in this format: x.x.x.x:port or [xx:xx:xx::x]:port host, _, err := net.SplitHostPort(stdout) if err != nil { // ginkgo.Fail the test if output format is unexpected. framework.Failf("exec pod returned unexpected stdout: [%v]n", stdout) } return pausePod.Status.PodIP, host } </del> // execAffinityTestForSessionAffinityTimeout is a helper function that wrap the logic of // affinity test for non-load-balancer services. Session afinity will be // enabled when the service is created and a short timeout will be configured so

What would you like to be : The test currently only works on Cloud Providers or environments that set the External IP on nodes, because it is using the framework directly to check the IPs of the pods, using a no-snat-test-proxy to check if the Pods are not using SNAT. However, this can be simplified, removing the no-snat-test-proxy Pod and executing the queries directly in a Pod. Why is this needed: The test should work in all environments, and it's failing https://k8s-,%20master&include-filter-by-regex=NoSNAT
/sig network /help wanted /failing test
/remove kind-feature /kind failing-test
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove kind-feature
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I would like to investigate this problem. /assign
I have filed , how could I verify if my patch fixes the problem?
I run them locally with KIND


Question: assert.IsType(t, core.GetActionImpl{}, actions[9]) } <ins> func TestUpdateNodeStatusAndVolumesInUseWithoutNodeLease(t *testing.T) { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.NodeLease, false)() cases := []struct { desc                  string existingVolumes       []v1.UniqueVolumeName // volumes to initially populate volumeManager existingNode          *v1.Node              // existing node object expectedNode          *v1.Node              // new node object after patch expectedReportedInUse []v1.UniqueVolumeName // expected volumes reported in use in volumeManager }{ { desc:         "no volumes and no update", existingNode: &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}}, expectedNode: &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}}, }, { desc:            "volumes inuse on node and volumeManager", existingVolumes: []v1.UniqueVolumeName{"vol1"}, existingNode: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ VolumesInUse: []v1.UniqueVolumeName{"vol1"}, }, }, expectedNode: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ VolumesInUse: []v1.UniqueVolumeName{"vol1"}, }, }, expectedReportedInUse: []v1.UniqueVolumeName{"vol1"}, }, { desc: "volumes inuse on node but not in volumeManager", existingNode: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ VolumesInUse: []v1.UniqueVolumeName{"vol1"}, }, }, expectedNode: &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}}, }, { desc:            "volumes inuse in volumeManager but not on node", existingVolumes: []v1.UniqueVolumeName{"vol1"}, existingNode:    &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}}, expectedNode: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ VolumesInUse: []v1.UniqueVolumeName{"vol1"}, }, }, expectedReportedInUse: []v1.UniqueVolumeName{"vol1"}, }, } for _, tc := range cases { t.Run(tc.desc, func(t *testing.T) { // Setup testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet kubelet.kubeClient = nil // ensure only the heartbeat client is used kubelet.containerManager = &localCM{ContainerManager: cm.NewStubContainerManager()} kubelet.lastStatusReportTime = kubelet.clock.Now() kubelet.nodeStatusReportFrequency = time.Hour kubelet.machineInfo = &cadvisorapi.MachineInfo{} // override test volumeManager fakeVolumeManager := kubeletvolume.NewFakeVolumeManager(tc.existingVolumes) kubelet.volumeManager = fakeVolumeManager // Only test VolumesInUse setter kubelet.setNodeStatusFuncs = []func(*v1.Node) error{ nodestatus.VolumesInUse(kubelet.volumeManager.ReconcilerStatesHasBeenSynced, kubelet.volumeManager.GetVolumesInUse), } kubeClient := testKubelet.fakeKubeClient kubeClient.ReactionChain = fake.NewSimpleClientset(&v1.NodeList{Items: []v1.Node{*tc.existingNode}}).ReactionChain // Execute assert.NoError(t, kubelet.updateNodeStatus()) // Validate actions := kubeClient.Actions() if tc.expectedNode != nil { assert.Len(t, actions, 2) assert.IsType(t, core.GetActionImpl{}, actions[0]) assert.IsType(t, core.PatchActionImpl{}, actions[1]) patchAction := actions[1].(core.PatchActionImpl) updatedNode, err := applyNodeStatusPatch(tc.existingNode, patchAction.GetPatch()) require.NoError(t, err) assert.True(t, apiequality.Semantic.DeepEqual(tc.expectedNode, updatedNode), "%s", diff.ObjectDiff(tc.expectedNode, updatedNode)) } else { assert.Len(t, actions, 1) assert.IsType(t, core.GetActionImpl{}, actions[0]) } reportedInUse := fakeVolumeManager.GetVolumesReportedInUse() assert.True(t, apiequality.Semantic.DeepEqual(tc.expectedReportedInUse, reportedInUse), "%s", diff.ObjectDiff(tc.expectedReportedInUse, reportedInUse)) }) } } func TestUpdateNodeStatusAndVolumesInUseWithNodeLease(t *testing.T) { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.NodeLease, true)() cases := []struct { desc                  string existingVolumes       []v1.UniqueVolumeName // volumes to initially populate volumeManager existingNode          *v1.Node              // existing node object expectedNode          *v1.Node              // new node object after patch expectedReportedInUse []v1.UniqueVolumeName // expected volumes reported in use in volumeManager }{ { desc:         "no volumes and no update", existingNode: &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}}, }, { desc:            "volumes inuse on node and volumeManager", existingVolumes: []v1.UniqueVolumeName{"vol1"}, existingNode: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ VolumesInUse: []v1.UniqueVolumeName{"vol1"}, }, }, expectedReportedInUse: []v1.UniqueVolumeName{"vol1"}, }, { desc: "volumes inuse on node but not in volumeManager", existingNode: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ VolumesInUse: []v1.UniqueVolumeName{"vol1"}, }, }, expectedNode: &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}}, }, { desc:            "volumes inuse in volumeManager but not on node", existingVolumes: []v1.UniqueVolumeName{"vol1"}, existingNode:    &v1.Node{ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}}, expectedNode: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ VolumesInUse: []v1.UniqueVolumeName{"vol1"}, }, }, expectedReportedInUse: []v1.UniqueVolumeName{"vol1"}, }, } for _, tc := range cases { t.Run(tc.desc, func(t *testing.T) { // Setup testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet kubelet.kubeClient = nil // ensure only the heartbeat client is used kubelet.containerManager = &localCM{ContainerManager: cm.NewStubContainerManager()} kubelet.lastStatusReportTime = kubelet.clock.Now() kubelet.nodeStatusReportFrequency = time.Hour kubelet.machineInfo = &cadvisorapi.MachineInfo{} // override test volumeManager fakeVolumeManager := kubeletvolume.NewFakeVolumeManager(tc.existingVolumes) kubelet.volumeManager = fakeVolumeManager // Only test VolumesInUse setter kubelet.setNodeStatusFuncs = []func(*v1.Node) error{ nodestatus.VolumesInUse(kubelet.volumeManager.ReconcilerStatesHasBeenSynced, kubelet.volumeManager.GetVolumesInUse), } kubeClient := testKubelet.fakeKubeClient kubeClient.ReactionChain = fake.NewSimpleClientset(&v1.NodeList{Items: []v1.Node{*tc.existingNode}}).ReactionChain // Execute assert.NoError(t, kubelet.updateNodeStatus()) // Validate actions := kubeClient.Actions() if tc.expectedNode != nil { assert.Len(t, actions, 2) assert.IsType(t, core.GetActionImpl{}, actions[0]) assert.IsType(t, core.PatchActionImpl{}, actions[1]) patchAction := actions[1].(core.PatchActionImpl) updatedNode, err := applyNodeStatusPatch(tc.existingNode, patchAction.GetPatch()) require.NoError(t, err) assert.True(t, apiequality.Semantic.DeepEqual(tc.expectedNode, updatedNode), "%s", diff.ObjectDiff(tc.expectedNode, updatedNode)) } else { assert.Len(t, actions, 1) assert.IsType(t, core.GetActionImpl{}, actions[0]) } reportedInUse := fakeVolumeManager.GetVolumesReportedInUse() assert.True(t, apiequality.Semantic.DeepEqual(tc.expectedReportedInUse, reportedInUse), "%s", diff.ObjectDiff(tc.expectedReportedInUse, reportedInUse)) }) } } </ins> func TestRegisterWithApiServer(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup()

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 2019-01-15 (when they were ) Testgrid link: Reason for failure: They all seem to be failing with: The failures seem to start with .
/sig storage
Summoning and This seems to have been failing since it was in master-blocking.
This is the same as
Thanks closing this one. /close
Closing this issue. <detailsIn response to : Thanks closing this one. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen I took a look at the latest test failures and I think it is a different issue than
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at one test failure here: Sequence of events: stopped, pod deleted: started again: volume to dsw started and failed because it's not in the node's volumesinuse list. This repeats for a while. also started, but we skipped cleaning up because the volume is in dsw so we want to let volume manager gracefully cleanup don't remove volume from dsw because actual state has not been updated? So it seems like there's a problem with populating the node's volumesinuse after a kubelet restart.
cc
/assign will continue investigating.

i will try repro-ing at without different commits to try to figure out which pr introduced the regression
/priority important-soon
I ran these tests against a cluster with the NodeLease feature disabled, and the tests pass. So I think the issue is related to the new NodeLease feature enabled a month ago: cc
I discussed with VolumesInUse is updated periodically in the Node.Status. So we need to make sure this flow is working well with the NodeLease feature.
and I walked through the code. The NodeLease feature introduced an early exit to the node status update function, so our theory is we need to add a call to . I will make the change and test it out


Question: }} } <ins> // NewGenerateNameConflict returns an error indicating the server // was not able to generate a valid name for a resource. func NewGenerateNameConflict(qualifiedResource schema.GroupResource, name string, retryAfterSeconds int) *StatusError { return &StatusError{metav1.Status{ Status: metav1.StatusFailure, Code:   http.StatusConflict, Reason: metav1.StatusReasonAlreadyExists, Details: &metav1.StatusDetails{ Group:             qualifiedResource.Group, Kind:              qualifiedResource.Resource, Name:              name, RetryAfterSeconds: int32(retryAfterSeconds), }, Message: fmt.Sprintf( "%s %q already exists, the server was not able to generate a unique name for the object", qualifiedResource.String(), name), }} } </ins> // NewUnauthorized returns an error indicating the client is not authorized to perform the requested // action. func NewUnauthorized(reason string) *StatusError {

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: job.Status.Failed = failed if err := jm.updateHandler(&job); err != nil { <del> return false, err </del> <ins> return forget, err </ins> } if jobHaveNewFailure && !IsJobFinished(&job) { // returning an error will re-enqueue Job after the backoff period <del> return false, fmt.Errorf("failed pod(s) detected for job key %q", key) </del> <ins> return forget, fmt.Errorf("failed pod(s) detected for job key %q", key) </ins> } forget = true

This issue was born out of the discussion from and In short: Job controller has an exponential backoff in place when failures happen to a job. As long as the parallelism is one, it's clear when a failure or success happens, the latter clears the backoff. The problem is when parallelism 1, which can lead to situation where the controller can observe successful and failed pod at the same time and won't clear the backoff leading to further retries and longer wait times (see for extensive details about the problem in e2e) Possible solutions: everything as is both success and failure are observed clear the backoff Pros and cons: Ad. 1. Aside from () I wonder if users got affected by it. Does it make sense to fix it at all? Ad. 2. This approach make sense in some situation, but the question arise, should that be controlled by a user. What if I have 9 failures and 1 success (parallelism = 10), should I still clear the backoff? Maybe we should introduce user controller field to set the failure rate in this case.
/cc
(jumping here 'cause forwarded me this). I personally think that the right approach is to reset the back-off as soon we have a successful pod run to complete. Back-off was introduced, at least in our case, to reduce the impact of bad configured PodSpec in Jobs. If in your run you do have some success, it means you've a good config and you just keep trying (not that different that what we did before ). I don't think it's worth asking user finer tune config. is already complex enough. My 2 cents.
cc
I agree with any success should immediately clear the backoff flag, since it shows the job is configured properly and the problem is only temporary. I'll assign it to myself, but anyone is open to submit a PR fixing this.
Hi I started to implement the pod success check in order to clear the backoff delay in case of at least one of the pods succeed. let me know if you are fine with this approach, and then I will go further with unit-tests and end2end tests.
is the rest of the work on this issue getting done for 1.10? Please update, and add approved-for-milestone if this is on track for 1.10.
I updated my change with missing unit-tests and created the PR:


Question: } func (f *fakeServerResources) ServerPreferredResources() ([]*metav1.APIResourceList, error) { <ins> f.Lock.Lock() defer f.Lock.Unlock() f.InterfaceUsedCount++ </ins> return f.PreferredResources, f.Error } <ins> func (f *fakeServerResources) setPreferredResources(resources []*metav1.APIResourceList) { f.Lock.Lock() defer f.Lock.Unlock() f.PreferredResources = resources } func (f *fakeServerResources) setError(err error) { f.Lock.Lock() defer f.Lock.Unlock() f.Error = err } func (f *fakeServerResources) getInterfaceUsedCount() int { f.Lock.Lock() defer f.Lock.Unlock() return f.InterfaceUsedCount } </ins> func (_ *fakeServerResources) ServerPreferredNamespacedResources() ([]*metav1.APIResourceList, error) { return nil, nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector.

When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.


Question: <ins> #!/bin/bash # Copyright 2015 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. mkdir -p /etc/google-fluentd/files if [ -z "$FILES_TO_COLLECT" ]; then exit 0 fi for filepath in $FILES_TO_COLLECT do filename=$(basename $filepath) cat > "/etc/google-fluentd/files/${filename}" << EndOfMessage <source> type tail format none time_key time path ${filepath} pos_file /etc/google-fluentd/fluentd-gcp.log.pos time_format %Y-%m-%dT%H:%M:%S tag file.${filename} read_from_head true </source> EndOfMessage done </ins>

We should provide a dockerfile for a pre-configured Fluentd image that can be passed file paths to collect logs from. We'll likely want one for exporting to elasticsearch and one for exporting to Google Cloud Logging, as we have today for the per-node stdout/stderr collectors. How to collect log files from within a container is a question that comes up occasionally (including ), and it provides a strong example of the usefulness of sidecar containers. cc
A controller that keeps one instance of a pod alive on each node would be useful for this. Shall I go ahead and write such a controller?
Would it be? This is intended to run as a sidecar container in a pod with the user's actual application. I think a pod-per-node controller is orthogonal (and is being discussed in and ). I'll create the actual Dockerfile, and a quick example of how to use it.
Sorry, my rush to catch up I think I mis-understood this thread.


Question: } } <ins> func NewTimeoutError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeTimeout, Key:                key, AdditionalErrorMsg: msg, } } </ins> func NewInvalidObjError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeInvalidObj,

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return c, informers } <del> func (f *fixture) runExpectError(deploymentName string) { f.run_(deploymentName, true) </del> <ins> func (f *fixture) runExpectError(deploymentName string, startInformers bool) { f.run_(deploymentName, startInformers, true) </ins> } func (f *fixture) run(deploymentName string) { <del> f.run_(deploymentName, false) </del> <ins> f.run_(deploymentName, true, false) </ins> } <del> func (f *fixture) run_(deploymentName string, expectError bool) { </del> <ins> func (f *fixture) run_(deploymentName string, startInformers bool, expectError bool) { </ins> c, informers := f.newController() <del> stopCh := make(chan struct{}) defer close(stopCh) informers.Start(stopCh) </del> <ins> if startInformers { stopCh := make(chan struct{}) defer close(stopCh) informers.Start(stopCh) } </ins> err := c.syncDeployment(deploymentName) if !expectError && err != nil {

From https://k8s- I've tried to reproduce locally in an endless loop but no failures so far. cc
Investigating.


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/errors" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/pkg/scheduler/algorithm"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: A failed Prioritize Map Function caused Scheduler break down. What you expected to happen: Pod remains Pending and describe pod will show errors of failed functions. How to reproduce it (as minimally and precisely as possible): It happens when some prioritize's map functions return with error. Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
As shown above, if priorityConfigs[i].Map(pod, meta, nodeInfo) returns with error, there will be no chance for results[i + 1] to initialize its value, thus causing panic when calling config.Reduce(pod, meta, nodeNameToInfo, results[i+1])
/sig scheduling
I opened a PR to initialize results with real host name instead of empty string.


Question: // Mount mounts the volume on the host. func (u *storageosUtil) MountVolume(b *storageosMounter, mntDevice, deviceMountPath string) error { <ins> klog.V(4).Infof("mounting StorageOS volume %q with namespace %q", b.volName, b.volNamespace) </ins> notMnt, err := b.mounter.IsLikelyNotMountPoint(deviceMountPath) if err != nil { if os.IsNotExist(err) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When a PVC is created and the PV mounted into a Pod using the StorageOS in-tree volume driver, the StorageOS CLI & UI do not report that the volume is mounted. The PV is created and mounted into the Pod successfully, and all k8s functionality works as expected. However, since the correct mount status is not visible to administrators using the StorageOS UI or CLI, they may make changes to or delete a mounted volume without realising, and the safeguards (must use to delete a mounted volume) do not trigger. This is a regression caused by , released in 1.14.0. What you expected to happen: The StorageOS CLI should report volume mount information (node name, mount timestamp). How to reproduce it (as minimally and precisely as possible): StorageOS node containers with CSI disabled: application Pod with a PVC using the StorageOS StorageClass: StorageOS to list volumes: Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.0 and later Cloud provider or hardware configuration: any OS (e.g: ): any linux (incl. Ubuntu 18.04) Kernel (e.g. ): any linux (incl. 4.15.0-33-generic) Install tools: any (incl. minikube) Network plugin and version (if this is a network-related bug): N/A Others: Requires StorageOS deployment /sig storage /assign


Question: The top command allows you to see the resource consumption for nodes or pods. <del> This command requires Heapster to be correctly configured and working on the server. `)) </del> <ins> This command requires Metrics Server to be correctly configured and working on the server. `)) </ins> ) func NewCmdTop(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command {

I am running k8s 1.15 and I see kubectl top --help is returning the following info: ...This command requires Heapster to be correctly configured and working on the server. As far as I know Heaspter is deprecated since 1.11. 1) Should the command help be updated i.e. replace Heapster with Metrics Server? 2) Will the command work if Metrics Server is deployed in the cluster? 3) Is kubectl top utility the easiest way to display (actual) resource usage info for pods and nodes? Any suggestions?
/sig cli
submitted a pr for this yes yes, or kubernetes dashboard


Question: "time" "github.com/google/go-cmp/cmp" <ins> "github.com/onsi/gomega" </ins> v1 "k8s.io/api/core/v1" eventsv1 "k8s.io/api/events/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

/kind failing-test /kind flake First seen in an unrelated PR, but also occurs in periodic jobs. /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign This feature is reverted once, and I fixed the flake, I'm cherry-pick that back again.


Question: klog.V(4).Infof("%s: performed write of new data to ts data directory: %s", w.logContext, tsDir) // (7) <del> if err = w.createUserVisibleFiles(cleanPayload); err != nil { klog.Errorf("%s: error creating visible symlinks in %s: %v", w.logContext, w.targetDir, err) return err } // (8) </del> newDataDirPath := filepath.Join(w.targetDir, newDataDirName) if err = os.Symlink(tsDirName, newDataDirPath); err != nil { os.RemoveAll(tsDir)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Creating a pod with a nested subpath volume will cause the pod to fail to start on Windows containerd nodes. What you expected to happen: The pods should start normally. How to reproduce it (as minimally and precisely as possible): Create the following pod on a Windows node with containerd: apiVersion: v1 kind: Pod metadata: annotations: e2e-test-privileged-psp name: pod-subpath-test-downwardapi spec: containers: args: mounttest --filecontentinloop=/test-volume --retrytime=20 image: claudiubelu/agnhost:2.21 name: test-container-subpath-downwardapi volumeMounts: mountPath: /test-volume name: test-volume subPath: downward/podname volumes: downwardAPI: defaultMode: 420 items: fieldRef: apiVersion: v1 fieldPath: path: downward/podname name: test-volume Anything else we need to know?: This issue is causing the following tests to fail on Windows Containerd nodes: [sig-storage] Subpath Atomic writer volumes should support subpaths with downward pod [LinuxOnly] [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with projected pod [LinuxOnly] [Conformance] The tests have the ` tag because we did not support Windows Containerd nodes at the time, and the tests still do not work for Windows Docker nodes. After a fix for this bug is merged, these tests will have to be renamed to enable them for Windows Containerd environments but not for Windows Docker environments. /sig windows Environment: Kubernetes version (use ): v1.21.0-alpha.0.493+ Cloud provider or hardware configuration: N/A OS (e.g: ): Windows Server 1809, Containerd Kernel (e.g. ): 10.0. N/A Build Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale