Question: return jitteredPeriod } <ins> // Backoff implements BackoffManager.Backoff, it returns a timer so caller can block on the timer for jittered backoff. // The returned timer must be drained before calling Backoff() the second time </ins> func (j *jitteredBackoffManagerImpl) Backoff() clock.Timer { <del> j.backoffTimer.Reset(j.getNextBackoff()) </del> <ins> backoff := j.getNextBackoff() if j.backoffTimer == nil { j.backoffTimer = j.clock.NewTimer(backoff) } else { j.backoffTimer.Reset(backoff) } </ins> return j.backoffTimer }

I noticed that this code: was printing which should not happen. Also occurs in wait.Forever. I suspect that in regressed this as that was the most recent major update and I only started seeing this on a recent apimachinery bump.
/sig api-machinery /assign as the reviewer
GitHub didn't allow me to assign the following users: zhan849. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery /assign as the reviewer Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
This is a correctness regression because the wait methods expect not to be run run more often than the period and may guard expensive behavior. Suggest a revert followed by a more rigorous test and review on merge.
thanks for reporting this, and I apologize about the regression I made. I did some quick test, and I am able to reproduce it occasionally, which indicate a race somewhere: Test: Failed output: Successful output? In the original code I checked in, I initialized the timer in the backoff manager construction time: However, after I changed to initializing the timer when is called, I'm not able to reproduce the race. The results indicated that the timer initialization is likely to be the place that caused the regression , and seems to be able to get fixed by change the way the timer gets initialized. However I'm unable to identify why it could cause such race the first time. Are you able to provide any insights? /cc
why are we initializing a timer with a zero duration? that looks related to this issue:
I see us resetting a timer we have no guarantee is stopped/expired:
dropped the information needed to accurately stop/reset the timer as documented
Hi thanks for providing the information! seems like the correct place to initialize the timer is in the "Backoff" method, since zero duration is problematic.

When I was putting up the interface, I documented that the timer will not be drained, and the reason is that returns a timer explicitly for user to block on that, so it might be confusing to block in itself Shall we change the impl to always drain the channel in backoff instead or relying on user to use it correctly? with this being said, will be changed to , which means that the will still rely on user's providing information about whether the channel is drained or not.
/cc thoughts? ^^
- sorry for that - we hit the bug with resetting timer not the first time, I should have caught that (I remember at least two other bugs with that). I'm not sure revert is a good option, because there were PRs that were chained on that: - this would have to be a very careful manual revert which I'm not sure it justifies the risk, given that 1.18 is already out. Currently the behavior is unpredictable, because if getNextbackoof() called for the first time is non-zero, it may or may not be ready immediately depending on the race. That is obviously bad. Initializing with infinity may solve this particular problem, but I need to think more about it to ensure that we don't have any problems in the later phase too...
thanks ! and sry again I did not read the timer's doc carefully enough but simply followed the Clock interface :( To fix, we can also initialize the timer to nil and create a fresh one when is first called, it's safe as this component is designed to run in single-threaded environment. But to ensure timer is drained before reset, we need to have user explicitly pass in "if timer is drained" into the to accurately decide if needs to drain. Or, is it ok to document the behavior and assume user will drain on before calling next ?
Very quick (I might have missed something) glance from the code suggests, that the only place we would need to change is actually BackoffUntil method. So passing the "if timer is drained" bit seems less error prone imho.
We should avoid changing the interface in a patch release.

I think initializing the timer in Backoff() and documenting the returned timer must be expired and drained before calling Backoff() again is the most compatible thing we can do:
Thanks and I'm also more inclined to Jordan's proposal, besides avoid interface change, if user are able to accurately pass "if timer is drained" info, it's fair to let user finish draining. do you have a PR already? of if not, I can make one, and add more tests if possible about this failure case.
I do not have a PR open


Question: } func (plugin *fcPlugin) ConstructVolumeSpec(volumeName, mountPath string) (*volume.Spec, error) { <del> fcVolume := &v1.Volume{ Name: volumeName, VolumeSource: v1.VolumeSource{ FC: &v1.FCVolumeSource{}, }, </del> <ins> // Find globalPDPath from pod volume directory(mountPath) // examples: //   mountPath:     pods/{podUid}/volumes/kubernetes.io~fc/{volumeName} //   globalPDPath : plugins/kubernetes.io/fc/50060e801049cfd1-lun-0 var globalPDPath string mounter := plugin.host.GetMounter(plugin.GetPluginName()) paths, err := mount.GetMountRefs(mounter, mountPath) if err != nil { return nil, err } for _, path := range paths { if strings.Contains(path, plugin.host.GetPluginDir(fcPluginName)) { globalPDPath = path break } } // Couldn't fetch globalPDPath if len(globalPDPath) == 0 { return nil, fmt.Errorf("couldn't fetch globalPDPath. failed to obtain volume spec") } arr := strings.Split(globalPDPath, "/") if len(arr) < 1 { return nil, fmt.Errorf("failed to retrieve volume plugin information from globalPDPath: %v", globalPDPath) } volumeInfo := arr[len(arr)-1] // Create volume from wwn+lun or wwid var fcVolume *v1.Volume if strings.Contains(volumeInfo, "-lun-") { wwnLun := strings.Split(volumeInfo, "-lun-") if len(wwnLun) < 2 { return nil, fmt.Errorf("failed to retrieve TargetWWN and Lun. volumeInfo is invalid: %v", volumeInfo) } lun, err := strconv.Atoi(wwnLun[1]) if err != nil { return nil, err } lun32 := int32(lun) fcVolume = &v1.Volume{ Name: volumeName, VolumeSource: v1.VolumeSource{ FC: &v1.FCVolumeSource{TargetWWNs: []string{wwnLun[0]}, Lun: &lun32}, }, } glog.V(5).Infof("ConstructVolumeSpec: TargetWWNs: %v, Lun: %v", fcVolume.VolumeSource.FC.TargetWWNs, *fcVolume.VolumeSource.FC.Lun) } else { fcVolume = &v1.Volume{ Name: volumeName, VolumeSource: v1.VolumeSource{ FC: &v1.FCVolumeSource{WWIDs: []string{volumeInfo}}, }, } glog.V(5).Infof("ConstructVolumeSpec: WWIDs: %v", fcVolume.VolumeSource.FC.WWIDs) </ins> } return volume.NewSpecFromVolume(fcVolume), nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently, FC plugin returns volume name and empty FCVolumeSource during ConstrutVolumeSpec during . However, in order to recreate newMounter from volumeSpec, volume information are lacking. As a result, filesystem volume's reconstruction always fails. What you expected to happen: Filesystem volume's reconstruction works properly, then a volume is unmounted and detached from kubelet node. How to reproduce it (as minimally and precisely as possible): (1) Create filesystem volume using FC plugin (2) Restart kubelet service which a pod with FC volume is running Anything else we need to know?: cc Environment: Kubernetes version (use ):Install tools: local-cluster-
/assign
/sig storage


Question: <del> id: blog-pod </del> <ins> apiVersion: v1beta3 </ins> kind: Pod <del> apiVersion: v1beta1 desiredState: manifest: version: v1beta1 containers: - name: git-sync image: proppy/git-sync imagePullPolicy: PullAlways env: - name: GIT_SYNC_REPO value: https://github.com/GoogleCloudPlatform/kubernetes.git - name: GIT_SYNC_DEST value: /git volumeMounts: - name: markdown mountPath: /git - name: hugo image: proppy/hugo imagePullPolicy: PullAlways env: - name: HUGO_SRC value: /src/contrib/git-sync/demo/blog - name: HUGO_BUILD_DRAFT value: 'true' - name: HUGO_BASE_URL value: example.com volumeMounts: - name: markdown mountPath: /src - name: html mountPath: /dest - name: nginx image: nginx volumeMounts: - name: html mountPath: /usr/share/nginx/html ports: - name: http-server containerPort: 80 volumes: </del> <ins> metadata: labels: name: blog name: blog-pod spec: containers: - name: git-sync image: gcr.io/google_containers/git-sync imagePullPolicy: Always volumeMounts: </ins> - name: markdown <del> source: emptyDir: {} </del> <ins> mountPath: /git env: - name: GIT_SYNC_REPO value: https://github.com/GoogleCloudPlatform/kubernetes.git - name: GIT_SYNC_DEST value: /git - name: hugo image: gcr.io/google_containers/hugo imagePullPolicy: Always volumeMounts: - name: markdown mountPath: /src - name: html mountPath: /dest env: - name: HUGO_SRC value: /src/contrib/git-sync/demo/blog - name: HUGO_BUILD_DRAFT value: "true" - name: HUGO_BASE_URL value: example.com - name: nginx image: nginx volumeMounts: </ins> - name: html <del> source: emptyDir: {} labels: name: blog </del> <ins> mountPath: /usr/share/nginx/html ports: - containerPort: 80 volumes: - name: markdown emptyDir: {} - name: html emptyDir: {} </ins>

This causes . In contrib/git-sync demo, the container voluntarily exits and gets recreated to "periodically" sync the git repo, but the pod is then reported as not ready, The end result is that the service targets at this pod does not get endpoints and thus not working. Thanks to for helping with the debugging.
The container was not running when kubelet reported the status, hence it was considered not ready. Because the containers in a pod share the same fate, the pod was not perceived as not ready. Having short-lived containers as a service endpoint may encounter problems like this. I am not really convinced that this is kubelet's problem. It sounds like should be a cron job...
Forgot to cc
Is it alright if I modify git-sync to make it run a for-loop?
sure, but it would still be nice to able to have a way to have background containers restarting periodically in a pod w/o failing the service.
, will it be useful if we let the user to config which containers will be considered by kubelet when it reports the readiness of the pod?
I am not sure kubelet restart is the right mechanism to run periodic tasks, and in this case, recreating the container isn't particularly efficient. FYI, there is a discussion for adding a controller for cron jobs in .
Thanks Let's forget about letting kubelet restart periodic tasks. In general, if some containers are not critical to a service, ignoring the status of these containers and keeping the service alive will increase the availability of the service.
we'd like to have sidecar containers that aren't critical to a pod's overall functioning. i.e. a sidecar which can crash or become unavailable without pulling the pod from the load balancer. In our case it is a sidecar which delivers some configuration to disk from a notoriously flakey service. We'd like the pod to stay up, the conf it needs is on disk. Should I file a separate issue for this?


Question: fmt.Errorf("Fake error"), true, 0, 3, 0, 0, 0, 1, 3, 0, 0, nil, "", }, <del> "failed pod": { </del> <ins> "failed + succeed pods: reset backoff delay": { </ins> 2, 5, 6, false, 0, <del> fmt.Errorf("Fake error"), false, 0, 1, 1, 1, </del> <ins> fmt.Errorf("Fake error"), true, 0, 1, 1, 1, </ins> 1, 0, 1, 1, 1, nil, "", }, <ins> "only new failed pod": { 2, 5, 6, false, 0, fmt.Errorf("Fake error"), false, 0, 1, 0, 1, 1, 0, 1, 0, 1, nil, "", }, </ins> "job finish": { 2, 5, 6, false, 0, nil, true, 0, 0, 5, 0,

This issue was born out of the discussion from and In short: Job controller has an exponential backoff in place when failures happen to a job. As long as the parallelism is one, it's clear when a failure or success happens, the latter clears the backoff. The problem is when parallelism 1, which can lead to situation where the controller can observe successful and failed pod at the same time and won't clear the backoff leading to further retries and longer wait times (see for extensive details about the problem in e2e) Possible solutions: everything as is both success and failure are observed clear the backoff Pros and cons: Ad. 1. Aside from () I wonder if users got affected by it. Does it make sense to fix it at all? Ad. 2. This approach make sense in some situation, but the question arise, should that be controlled by a user. What if I have 9 failures and 1 success (parallelism = 10), should I still clear the backoff? Maybe we should introduce user controller field to set the failure rate in this case.
/cc
(jumping here 'cause forwarded me this). I personally think that the right approach is to reset the back-off as soon we have a successful pod run to complete. Back-off was introduced, at least in our case, to reduce the impact of bad configured PodSpec in Jobs. If in your run you do have some success, it means you've a good config and you just keep trying (not that different that what we did before ). I don't think it's worth asking user finer tune config. is already complex enough. My 2 cents.
cc
I agree with any success should immediately clear the backoff flag, since it shows the job is configured properly and the problem is only temporary. I'll assign it to myself, but anyone is open to submit a PR fixing this.
Hi I started to implement the pod success check in order to clear the backoff delay in case of at least one of the pods succeed. let me know if you are fine with this approach, and then I will go further with unit-tests and end2end tests.
is the rest of the work on this issue getting done for 1.10? Please update, and add approved-for-milestone if this is on track for 1.10.
I updated my change with missing unit-tests and created the PR:


Question: record_command run_wait_tests <ins> #################### # kubectl debug    # #################### if kube::test::if_supports_resource "${pods}" ; then record_command run_kubectl_debug_pod_tests fi if kube::test::if_supports_resource "${nodes}" ; then record_command run_kubectl_debug_node_tests fi </ins> cleanup_tests }

What would you like to be : Add integration tests for Why is this needed: This is part of the kubernetes/enhancements /sig cli /priority important-soon
/assign
/triage accepted


Question: }, Spec: v1.PodSpec{ Containers:        []v1.Container{container}, <ins> Priority:          &priority, </ins> PriorityClassName: "system-node-critical", HostNetwork:       true, Volumes:           VolumeMapToSlice(volumes),

etcd / kube-apiserver stops before most containers, despite using . On a single node server, with multus (and likely other CNI talking to the API), this spam the logs and make the shutdown slower should stop one group at a time On a single node cluster, have a slow to shutdown pod (sleep inf), look at the logs to see in which order they are killed Reading the kep , the summary says so I would expect all pods with priority between 0 and to be stopped first, then coredns (), then the rest. This is definitely not what's happening: <details</detailsN/A <details</detailskubeadm containerd multus
/cc /sig node
Each will match all Pods whose is less than or equal to it. This may suit your want
/kind support
sorry for the delay I missed the notification ... this is the opposite of what the documentation says :( Maybe i'm misunderstanding the feature, I want shutdown ordering, and it's not working for me. All my pods have priority set to either 0, or . I want all the pods with priority 0 to shutdown first, once they are all shutdown or grace period is elapsed, shutdown the ones with priority , wait for them to shutdown or grace period, and finally take care of the ones with priority . Here some failing tests: test1: First killed pod was etcd (verified priority is ) ... test2: first pod killed was kube-scheduler test3 (same as test2): first pod killed was kube-controller-manager test4: first pod killed was etcd test5 (config in reverse order as test1): etcd was killed 3rd but still before pods with priority 0
I see what you mean, but this shouldn't be happening. It should only shutdown based on the priority of the group from lowest to highest. I would like to know the priority of all Pods on the node you shut down, and the configuration of , to try to reproduce it on my side
I'm reproducing it on an empty cluster, but just noticed coredns is killed correctly, it's just all containers from last group end up in first group
For some reasons the system-node-critical / pods go with the priority 0 pods
I think the pods quit too quickly so it looks like they were deleted together, maybe you should block them until the last moment.

I think I found a lead, it's all the static pods that are killed too soon, not kubeproxy () they have priorityClassName set, but not priority, so maybe priority is nil when getting of a static pod
I see that in numeric priorities to manifests in cluster/gce/manifests/ with the comment in the PR , are priority mandatory for static pods ? should we make kubeadm add them, or fix that in kube somewhere ?
This is the result from the API, I bet kubelet has different path to get pod manifests for cluster pods and static pods, as kubelet can work without the API
If I manually add to etcd manifest it's killed in the last group with kubeproxy
/remove-kind support
/triage accepted
[For a single-node cluster], I wonder if we should actually set a higher priority than for all or a subset of control plane Pods, because even with this change we could still shut down the API server quickly and before a node-critical component has completed its own shutdown.
I agree with this idea, because of the dependencies between the components of node-critical.


Question: zoneSizeKey             = "zone_size" zoneNoUnhealthyNodesKey = "unhealthy_nodes_in_zone" evictionsNumberKey      = "evictions_number" <ins> evictionsTotalKey       = "evictions_total" </ins> ) var (

I would like to propose the graduation of the metric from alpha to stable. The metric has been stable for 2 years now.
/sig api-machinery /sig instrumentation
I am willing to do all StabilityLevels from alpha to stable? Is that right?
/assign
- The request is to have the stability level for the metric graduate to stable. If there are other metrics you want to do at the same time that's a bonus, but not part of the request in this issue.
/sig node /remove-sig api-machinery /cc
To be promoted to stable, the metrics would need to be renamed from to in order to follow Prometheus best practices as well as sig-instrumentation naming guidelines.
/cc
/assign
/triage accepted


Question: containersToKeep := p.containersToKeep if removeAll { containersToKeep = 0 <ins> filterContainerID = "" </ins> } for _, candidate := range getContainersToDeleteInPod(filterContainerID, podStatus, containersToKeep) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: a statefulset which will never be running: down it the pods seems like only running pods will be deleted. And CrashLoopBackOff pods are not included into "CURRENT" pod. What you expected to happen: I guess the failss-0 should be deleted after replicas set to 0, but it doesn't. Is such behavior designed by purpose? How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Change to , and it can be scaled down... Now the crashloopbackoff pods got deleted... Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): debian Kernel (e.g. ): Install tools: Others:
Update: Cannot scale down to 0, if the pods keep . How to reproduce it any statefulset whose PVC cannot be bound. the replicas to 0
/sig apps because statefulset
I'll have a look at this issue
I'm afraid it requires bigger code/design change because controller first updates its pods/replicas and when every replica is and then it starts to terminating redundant pods (scaling down). What do you think about changing the order? I mean scale down first and then update every StatefulSet's pod. For more details take a look here:
I think when set the replicas to 0, all pending or crashed pods should be included in the array and then removed.
I've observed something similar as well - I think it may be a Kubelet issue though (or there are two issues together), because what I see is: a stateful set size 1 with a pod that crash loops after 5-10s the pod directly pod is in , pod is deleted quickly pod is in , pod takes tens of seconds to be deleted My suspicion is that there is a node level issue as well where the pod worker on the kubelet is interacting with crash loop backoff incorrectly. was seeing this in 1.9
I just tried this with a bare pod and could not reproduce on master.

create pod that immediately fails wait for cashloop backoff to reach 5m attempt to delete the pod In my case, the pod deleted immediately, even though it had just started the 5m backoff interval. So this is either fixed already in master (1.10) there is some element of the recreate that is not captured when creating bare pods
Re-reading through the issue, seems like there is some element of Stateful sets at play here that does not effect bare pods or ReplicationControllers.
I was able to recreate with a StatefulSet. He is the kubelet + controller-manager log for the time between the pod delete request and the StatefulSet controller creating the new one (with a 10s yet unexplained delay) There are a number of that seem to be holding up the deletion of the pod. However, the the REMOVE comes in from the API server 10s after the first DELETE forces the pod the cleanup and the StatefulSet to start a new pod.
Nevermind the part about not being able to create it with a bare pod. I can. I forgot I set a termination grace period of 0 on the pod and the issue is related to the grace period. What happens is that the kubelet is not cleaning up the failed container and thus the pod is not cleaned up and not being completely deleted. If there are any containers that still exist in the pod, then the delete can't be finalized. Thus, the delete waits until the grace period is over and then the pod is removed from the API server causing the kubelet to clean up along another path. Still running down where this should be cleaned up.
/sig node
Hi, I am also encountering this issue using StatefulSet on 1.8. Is there any known workaround we can use in the short term?


Question: return host, nil } <ins> // GetAddressesByTypeAndFamily returns a list of addresses of the given addressType for the given node // and filtered by IPFamily func GetAddressesByTypeAndFamily(node *v1.Node, addressType v1.NodeAddressType, family v1.IPFamily) (ips []string) { for _, nodeAddress := range node.Status.Addresses { if nodeAddress.Type != addressType { continue } if nodeAddress.Address == "" { continue } if family == v1.IPv6Protocol && netutil.IsIPv6String(nodeAddress.Address) { ips = append(ips, nodeAddress.Address) } if family == v1.IPv4Protocol && !netutil.IsIPv6String(nodeAddress.Address) { ips = append(ips, nodeAddress.Address) } } return } </ins> // GetAddresses returns a list of addresses of the given addressType for the given node func GetAddresses(node *v1.Node, addressType v1.NodeAddressType) (ips []string) { for j := range node.Status.Addresses {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-network-kind-ipvs,dual,master sig-network-kind-ipvs,IPv6,master Which test(s) are failing: [x] KubeProxy should set TCP CLOSE_WAIT timeout [x] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort Service [x] KubeProxy should resorve connection reset issue (tracking in ) Since when has it been failing: Oct 28 Testgrid link: sig-network-kind-ipvs,dual,master  Reason for failure: Both appear to be failing on missing conntrack entries ex. Anything else we need to know:
/sig network /cc /assign
I'm spinning up a kind cluster with ipv6+ipvs to see if I can replicate this test locally
/triage accepted
heh, the job was not running it, it started to run when we changed the tag I meant this test and the PR where we renamed the kube-proxy test
this one has an issue already open, and I have a PR for fixing it
reached out to me on Slack and we identified as the high priority. I have tested this locally and it fails on both IPTables and IPVS kind clusters locally. I'm digging through the cluster logs now.
I' ve logging to conntrack with level v4 recently just for these problems, try to bump your kube-proxy logging to v4
seems that there are less test failures on the IPv6 jobs, let's keep watching them :eyes: However, the dual-stack jobs are legit failures, basically because on dualstack clusters, the node has 2 internal IPs, and those tests use the internal IP for asserting some operations, and the problem is that they are picking the internalIP of the wrong IP family, hence the test fails The solution should be to add the InernalIp correspoding to the cluster primary IP familty, you can obtain it with but it seems that we have to modify the function or add a new one that returns the nodes and the corresponding ips
Hey Thanks for all the help, will look into the areas of the code and try to bump the logging.


Question: klog.V(5).Infof("Installing health checkers for (%v): %v", path, formatQuoted(checkerNames(checks...)...)) <del> mux.Handle(path, handleRootHealthz(checks...)) </del> <ins> mux.Handle(path, metrics.InstrumentHandlerFunc("GET", /* group = */ "", /* version = */ "", /* resource = */ "", /* subresource = */ path, /* scope = */ "", /* component = */ "", handleRootHealthz(checks...))) </ins> for _, check := range checks { mux.Handle(fmt.Sprintf("%s/%v", path, check.Name()), adaptCheckToHandler(check.Check)) }

What would you like to be : Currently the apiserver has request latencies metrics, which are a histogram of requests to the apiserver. However, these request metrics do not include healthz. While we have recently the ability to get back health check metrics from the healthz endpoint directly, we should also emit this data as a metric, so that we can monitor health check latencies. /assign /sig api-machinery instrumentation
GitHub didn't allow me to assign the following users: jktomer. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /sig api-machinery instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I noticed this is informally assigned. Has there been any progress? Is this a place I can fit in?
Yes, I'm working on a PR.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } return client } <ins> // DiscoveryClientOrDie returns a discovery.DiscoveryInterface built from the ClientBuilder // Discovery is special because it will artificially pump the burst quite high to handle the many discovery requests. func (b SimpleControllerClientBuilder) DiscoveryClient(name string) (discovery.DiscoveryInterface, error) { clientConfig, err := b.Config(name) if err != nil { return nil, err } // Discovery makes a lot of requests infrequently.  This allows the burst to succeed and refill to happen // in just a few seconds. clientConfig.Burst = 200 clientConfig.QPS = 20 return clientset.NewForConfig(clientConfig) } // DiscoveryClientOrDie returns a discovery.DiscoveryInterface built from the ClientBuilder with no error. // Discovery is special because it will artificially pump the burst quite high to handle the many discovery requests. // If it gets an error getting the client, it will log the error and kill the process it's running in. func (b SimpleControllerClientBuilder) DiscoveryClientOrDie(name string) discovery.DiscoveryInterface { client, err := b.DiscoveryClient(name) if err != nil { klog.Fatal(err) } return client } </ins>

Ran kube-controller-manager with and observed this in the log every 30 seconds: This is also happening in 1.20, 1.19, and 1.18 The GC and quota clients look up discovery information every 30 seconds. They have a QPS of 20 requests per second and a burst of 30 requests. Default configuration of kube-controller-manager runs without constant throttling warnings. /sig api-machinery
/assign /triage accepted


Question: - --memory={{ base_metrics_memory }} - --extra-memory={{ metrics_memory_per_node }}Mi - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=heapster - --poll-period=300000 - --estimator=exponential

I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue?

This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.


Question: optional string resourceVersion = 4; // Timeout for the list/watch call. <ins> // This limits the duration of the call, regardless of any activity or inactivity. </ins> // +optional optional int64 timeoutSeconds = 5;

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Watches done with the field of set to some value T end after T regardless of whether there has been any activity in the meantime. What you expected to happen: I expected the watch to end when no events have been delivered in the last T. In ordinary usage, a "timeout" is something that happens if there has been no activity for the specified length of time. How to reproduce it (as minimally and precisely as possible): Create a configmap named in namespace . Then in one shell thrash it periodically like this: Observe that it proceeds to update the configmap every 7 seconds or so. In another shell do a watch with a timeout; observe that it ends even though the thrashing continues. Anything else we need to know?: Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.4+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-21T17:22:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}Cloud provider or hardware configuration: local hardware OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): Linux n17s2ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: local Others:
I am told the observed behavior is the intended behavior, the problem here is that the field name and comment are misleading.
/sig api-machinery
It's a timeout on the request lifetime, not an inactivity timeout
The inactivity condition is implied in ordinary usage of the word "timeout". A word like "lifetime", "duration", or "time limit" would be a better description of the existing behavior. I do not propose changing the field name for this, that would be more work than it is worth. I propose updating the commentary on that field to make its meaning clear. BTW, what exactly is this field's meaning to a list operation?


Question: // RuleWithOperations is a tuple of Operations and Resources. It is recommended to make // sure that all the tuple expansions are valid. message RuleWithOperations { <del> // Operations is the operations the admission hook cares about - CREATE, UPDATE, or * // for all operations. </del> <ins> // Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * // for all of those operations and any future admission operations that are added. </ins> // If '*' is present, the length of the slice must be one. // Required. repeated string operations = 1;

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Working on other doc bugs, I went looking for what calls admission plugins. I found four call sites for validating plugins, documented in . They correspond to the verbs CONNECT, CREATE, DELETE, and UPDATE. But only two of those verbs are documented at What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Looks like docs are correct in and
The API doc can be updated to call out all four operations and clarify that "all" means "all four admission operations"
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/assign


Question: preemptionStartTime := time.Now() defer func() { metrics.PreemptionAttempts.Inc() <del> metrics.SchedulingAlgorithmPreemptionEvaluationDuration.Observe(metrics.SinceInSeconds(preemptionStartTime)) </del> <ins> metrics.DeprecatedSchedulingAlgorithmPreemptionEvaluationDuration.Observe(metrics.SinceInSeconds(preemptionStartTime)) </ins> }() nnn, err := pl.preempt(ctx, state, pod, m)

Mark the following metrics as deprecated and . Both of those metrics are now covered as part of , the former as a the latter and a plugin. This can be done by setting in each of the two metrics opts. Changing the name of the metric variable to be prefixed with "Deprecated" is useful as well. /help /sig scheduling
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Mark the following metrics as deprecated and . Both of those metrics are now covered as part of , the former as a the latter and a plugin. /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/kind deprecation


Question: } func TestOpGet(t *testing.T) { <del> simpleStorage := &SimpleRESTStorage{} </del> <ins> testOver := make(chan struct{}) defer close(testOver) simpleStorage := &SimpleRESTStorage{ injectedFunction: func(obj interface{}) (interface{}, error) { // Eliminate flakes by ensuring the create operation takes longer than this test. <-testOver return obj, nil }, } </ins> handler := New(map[string]RESTStorage{ "foo": simpleStorage, }, codec, "/prefix/version")

https://travis- Seems surprising-- I will look into it later if no one else does first.
Got another flake. Looking into it.


Question: Capacity: v1.ResourceList{ v1.ResourceName(v1.ResourceStorage): resource.MustParse(fmt.Sprintf("%dKi", volSpec.Size)), }, <ins> VolumeMode: volumeMode, </ins> PersistentVolumeSource: v1.PersistentVolumeSource{ VsphereVolume: &v1.VsphereVirtualDiskVolumeSource{ VolumePath:        volSpec.Path,

FEATURE REQUEST: vSphere volume plugin does not support volume device yet. Need to add volumeMode in PV spec as other plugin did. release 1.12
/sig storage


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> FROM BASEIMAGE as builder </del> <ins> ARG BASEIMAGE ARG RUNNERIMAGE FROM ${BASEIMAGE} as builder </ins> # This image needs bash for running "migrate-if-needed.sh". Instead of a full debian image # we use just the bash-static and we wrap bash-static into a distroless image instead of

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Building the etcd integration-test-image fails with the following err: Failure Log: Expected the image build to pass. k/k cluster/images/etcd build-integration-test-image Change that probably broke it: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig etcd
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig etcd Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
/sig api-machinery
/cc /cc
/cc
/assign /triage accepted
is working on this in .


Question: pod:    podWithClaimName, claims: []*resourcev1alpha2.ResourceClaim{pendingDelayedClaim}, want: want{ <del> filter: perNodeResult{ workerNode.Name: { status: framework.AsStatus(fmt.Errorf(`look up resource class: resourceclass.resource.k8s.io "%s" not found`, className)), }, </del> <ins> prefilter: result{ status: framework.AsStatus(fmt.Errorf(`look up resource class: resourceclass.resource.k8s.io "%s" not found`, className)), </ins> }, postfilter: result{ <del> status: framework.NewStatus(framework.Unschedulable, `still not schedulable`), </del> <ins> status: framework.NewStatus(framework.Unschedulable, `no new claims to deallocate`), </ins> }, }, },

The node selectors gets parsed by in for each node. That work can be done once in . Better performance.
/sig node /sig scheduling /triage accepted /priority backlog
I can try this one. /assign


Question: pvc := createPVC("fake-pvc", "5G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}) <del> w, _ := testClient.PersistentVolumes().Watch(api.ListOptions{}) </del> <ins> w, err := testClient.PersistentVolumes().Watch(api.ListOptions{}) if err != nil { t.Errorf("Failed to watch PersistentVolumes: %v", err) } </ins> defer w.Stop() <del> _, _ = testClient.PersistentVolumes().Create(pv) _, _ = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) </del> <ins> _, err = testClient.PersistentVolumes().Create(pv) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } </ins> // wait until the controller pairs the volume and claim waitForPersistentVolumePhase(w, api.VolumeBound)

PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.


Question: } // Currently two modes of proxying are available: 'userspace' (older, stable) or 'iptables' <del> // (newer, faster). If blank, look at the Node object on the Kubernetes API and respect the // 'net.experimental.kubernetes.io/proxy-mode' annotation if provided.  Otherwise use the // best-available proxy (currently iptables, but may change in future versions).  If the // iptables proxy is selected, regardless of how, but the system's kernel or iptables // versions are insufficient, this always falls back to the userspace proxy. </del> <ins> // (newer, faster). If blank, use the best-available proxy (currently iptables, but may // change in future versions).  If the iptables proxy is selected, regardless of how, but // the system's kernel or iptables versions are insufficient, this always falls back to the // userspace proxy. </ins> type ProxyMode string const (

This annotation dates back almost a year. This is a relatively easy project for someone to contribute.
I'll work on getting a pull request in tomorrow for this. Ive been looking at contributing and think this would be a great place to start
nevermind. Looks like a pull request already exists. Overlooked the mention due to being on mobile. Cheers
We'll file more :)


Question: }, expectedTrace: []*spanExpectation{ { <del> name: "KubernetesAPI", </del> <ins> name: "GET /api/v1/nodes/{:name}", </ins> attributes: map[string]func(*commonv1.AnyValue) bool{ "http.user_agent": func(v *commonv1.AnyValue) bool { return strings.HasPrefix(v.GetStringValue(), "tracing.test")

When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/rest" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/strategicpatch"

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.


Question: }}, AddressType: addressType, Endpoints: []discoveryv1.Endpoint{ <del> {Addresses: []string{pod.Status.PodIP}}, </del> <ins> { Addresses:  []string{pod.Status.PodIP}, Conditions: discoveryv1.EndpointConditions{Ready: &readyCondTrue}, }, </ins> }, }

The test create custome EndpointSlices to validate some behaviors sending traffic, so it is clear that it expects the EndpointSlices to be ready, The API doc is vague about this situation and leaves room to interpret the "unknown behavior", quoting Since the test expects the EndpointSlices to be Ready we should set them, if we decide to change the API behavior, then we should add an additional e2e test to validate this behavior, but not use the same test to validate two different things /kind bug /kind failing-testing /sig network Slack thread
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : The test create custome EndpointSlices to validate some behaviors sending traffic, so it is clear that it expects the EndpointSlices to be ready, /kind failing-testing /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: aauren. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks I'll take this one and draft up a PR this weekend. (this comment should hopefully allow the k8s ci robot to assign this to me as well)
if you are planning to contribute more often you should apply as org member


Question: uuid := utilstrings.ShortenString(id.uuid, 8) switch reason { case "Created": <del> r.recorder.Eventf(ref, v1.EventTypeNormal, events.CreatedContainer, "Created with rkt id %v", uuid) </del> <ins> r.recorder.Eventf(events.ToObjectReference(ref), v1.EventTypeNormal, events.CreatedContainer, "Created with rkt id %v", uuid) </ins> case "Started": <del> r.recorder.Eventf(ref, v1.EventTypeNormal, events.StartedContainer, "Started with rkt id %v", uuid) </del> <ins> r.recorder.Eventf(events.ToObjectReference(ref), v1.EventTypeNormal, events.StartedContainer, "Started with rkt id %v", uuid) </ins> case "Failed": <del> r.recorder.Eventf(ref, v1.EventTypeWarning, events.FailedToStartContainer, "Failed to start with rkt id %v with error %v", uuid, failure) </del> <ins> r.recorder.Eventf(events.ToObjectReference(ref), v1.EventTypeWarning, events.FailedToStartContainer, "Failed to start with rkt id %v with error %v", uuid, failure) </ins> case "Killing": <del> r.recorder.Eventf(ref, v1.EventTypeNormal, events.KillingContainer, "Killing with rkt id %v", uuid) </del> <ins> r.recorder.Eventf(events.ToObjectReference(ref), v1.EventTypeNormal, events.KillingContainer, "Killing with rkt id %v", uuid) </ins> default: glog.Errorf("rkt: Unexpected event %q", reason) }

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): kubernetes/master using CRI with docker What happened: Create a pod that uses a readiness probe that does an exec to command that doesnt exist See the following in logs: What you expected to happen: No error in log, expected the event to properly be thrown. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know: Nope.
this is a regression.
debugging, will post a fix once found.
Other reports: ,
-- thanks for the other reports. at least now we have a reliable reproduction.
The problem appears to be the following: has a that correlates containers to a . It uses the type . event recorder library we use in the prober works with . If it doesn't get one of those, it expects the passed in object to implement List. As a result, we are passing a v1.ObjectReference the recorder was not expecting.
The bug may have been introduced in , which was there since v1.6.0.
Opened for a fix.
Keep this open until 1.6 is patched.
/remove-priority P0 /priority critical-urgent (I'm not actually sure this is the right priority, just trying to remove the old priority/PN labels)
Closing since has been merged.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 6 tree at Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Ubuntu xenial Kernel (e.g. ): Linux fab7-compute-2 4.4.0-70-generic -Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm What happened: This gets logged ] Could not construct reference to: '&v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns--72wvm", UID:"-1edc-11e7-ac00-", APIVersion:"v1", ResourceVersion:"554", FieldPath:"spec.containers{kubedns}"}' due to: 'object does not implement the List interfaces'. Will not report event: 'Warning' 'Unhealthy' 'Readiness probe failed: Get http://10.1.0.130:8081/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)' What you expected to happen: Should log correctly How to reproduce it (as minimally and precisely as possible): Create pod that fails readiness probe
cc not sure if this is related to your changes
I don't think so. I didn't change the identification of or which are being used there.

I always get the same message. What does this exactly mean?
We're also seeing this on 1.6.1
Seeing this in as well
How we can come to know which release this is fixed? I am using 1.6.6 still I can see this issue. Thanks in Advance.


Question: examplev1 "k8s.io/apiserver/pkg/apis/example/v1" "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/registry/rest" <ins> clientgoscheme "k8s.io/client-go/kubernetes/scheme" </ins> utiltrace "k8s.io/utils/trace" )

What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: PredicateToConfigProducer: make(map[string]ConfigProducer), PriorityToConfigProducer:  make(map[string]ConfigProducer), } <ins> // Register Predicates. </ins> registry.RegisterPredicate(predicates.GeneralPred, func(_ ConfigProducerArgs) (plugins config.Plugins, pluginConfig []config.PluginConfig) { // GeneralPredicate is a combination of predicates.

What would you like to be : Add a filter plugin that calls into EvenPodsSpread predicate, example Why is this needed: Part of /sig scheduling /help /priority important-soon
/assign
/remove-help
/retitle: [migration phase 1] EvenPodsSpread as filter plugin
/retitle [migration phase 1] EvenPodsSpread as filter plugin
/assign /unassign
May I take over this?
is going to take this over after the offline discussion, and he is familiar with EvenPodsSpread
IC, thanks a lot.


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> "k8s.io/apimachinery/pkg/labels" </ins> "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/discovery" clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/util/retry" </ins> apiregistrationv1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" aggregatorclient "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset" rbacv1helpers "k8s.io/kubernetes/pkg/apis/rbac/v1" "k8s.io/kubernetes/test/e2e/framework" <ins> e2eauth "k8s.io/kubernetes/test/e2e/framework/auth" </ins> e2edeployment "k8s.io/kubernetes/test/e2e/framework/deployment" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image"

[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below.

This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: } _, ok := staged[tc.volID] <del> if ok && tc.stageUnstageSet { </del> <ins> if ok && tc.stageUnstageSet && tc.volID != "" { </ins> t.Errorf("found unexpected staged volume: %s", tc.volID) } else if !ok && !tc.stageUnstageSet { t.Errorf("could not find expected staged volume: %s", tc.volID)

Hello, I've recently been testing the reliability of storage for k8s v1.25.0. A strange phenomenon occurs occasionally during the test: the volume cannot be detached after the PVC and pod are deleted(the reclaimPolicy of pv is delete. So I first went to check the corresponding and objects and found that they both existed. Then I checked the kube-controllerManager and kubelet logs and found that kubelet logs kept printing: But I found that at the beginning, the volume was unmountDevice succeed, as shown in the following kubelet log: it is so wired ! Then I some log variables to the unmountDetachDevices method: The results show that And I've drawn up a time-series chart of the contents of the log. ! Let's go back to the unmountDetachDevices method. According to the above-mentioned log content, the first operation successfully changes of the corresponding attachedVolume to , but the function in the subsequent operation still uses stale DeviceMountState. that the unmountDevice operation is not performed after the first unmountDevice operation succeeds. multiple unmountDevice operations are allowed, ensure that the correct pvname can be obtained in the method. the volume is detached after the pod and PVC objects are deleted This problem can be reproduced by repeatedly deleting and recreating pods that require PVC in k8s 1.25.0 cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage
we can't safely assume that the json is always there.
ping
Thanks for the detailed analysis . I'm still sorting through this, do you think that is involved with this?
Yes. look at this log content: The global mount path is changed in the As a result, the method obtains incorrect pvName(e6104b10fea107e004e3307b0444fd7b775fedd804ab47aa997118b7927c32e0). The unmountDevice operation may be performed multiple times in [1.19 - 1.25] kubernetes versions. This problem does not occur before because can still obtain the correct pvName from the global mount path. However, this problem occurs after is merged. https://user- <br</brAfter the first unmountDevice operation is performed successfully, the file is deleted.

So the subsequent unmountDevice operation will perform method because file doesn't exist.
Awesome, thanks. Let me find someone to work on this.
I ll take a look at the issue, since its related to the change I made Note: this change needs the node to drained before upgrading to a version with this change . Please check the release notes
Thanks a lot !
yep, I always reproduce this problem in a newly created k8s 1.25.0 cluster.
Hihas there been any progress on this issue?
/priority important-soon
/triage accepted
Hihas there been any progress?
I also hit this error after pvc and pod deleted on k8s 1.25: and then these two volumes cannot be unmounted on the node forever: I tried to delete pv, it's stuck in terminating state forever
restart the kubelet on the node in problem could mitigate this issue, but we really need to fix this issue ASAP, I could repro this issue easily when running 1000 disk attach/detach test
I think this is again a TOCTOU bug in unmountDetachDevices.
The Unmount operation is , thus the state may not be set as before next reconcile, so it is possible that unmountDevice operation performed multiple times. So the root cause of this issue is that got wrong PV name, since changed global mount path.
I have found a similar issue in our environment, which is based on Kubernetes 1.25.3. If I install a Pod with a PVC, and I force move the Pod with NodeAffinity around the cluster, around 10-20 retries the Pod does not come up: the VolumeAttachment is bound to the old node and is not moved by the system anymore. If this is the same issue as reported here I do not know, but the logs are similar: From the Worker's kubelet where the volumeattachment is bound to: Also I see lots of these: But, if this is the same, I would like to know that the referenced PR was merged into exactly which Kubernetes version, as I have found in the RELEASE NOTES, that it was in 1.24.0-alpha: But we could not reproduce the issue on Kubernetes 1.24.2. How come?
As per the logs, it's the same issue. Kubelet operates in a reconcile loop, and is an async operation.

This bug only happened when is operated for multiple times, so it cannot be 100% reproduced. However, if the possibility of reproduction in 1.25 is much higher than the cluster lower than 1.25, we should figure out whether there are some other changes that makes this bug happened more frequently.
I have 1K disk attach/detach to 40 nodes test, if it runs multiple times, it could 100% repro, we could use this patch for testing, if 100 test runs pass, then it could prove the fix really works.
Sure. I mean not every single Mount/Unmount operation will lead to this error, but if we do it enough times, then it could be 100% to reproduce.
May I ask what could be a workaround in production systems or even CI systems on this issue? Our CI is struggling with this and other random issues, to the point we would like to stabilize it by kicking the system in a way to fix things... I was trying a kubelet restart on the Worker where the Mount is stuck... but many cases the kubelet restart itself hangs or does not produce a correct restart. So what could be a workaround if we find out that there is a hung volumeattachment?
restart kubelet, and if restart kubelet hangs, reboot the node.
Volumeattachment failed to detach from old node When statefulset application with pvc scheduled on worker1 during deployment its volumeattachment bounds to worker1 when same application upgraded scheduled on another worker node say worker2 and struck in ContainerCreating state and volumeattachment failed to detach from old node .

when pod described: Events: Warning FailedMount 23m (x820 over 6d2h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[ datadir]: timed out waiting for the condition Warning FailedMount 5m22s (x3069 over 6d3h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition Environment: NAME="SLES" VERSION="15-SP4" VERSIONID="15.4" PRETTYNAME="SUSE Linux Enterprise Server 15 SP4" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp4" DOCUMENTATIONURL=""' Kubernetes version: v1.25.3 kernel: 5.14.21-.24.41-default Storage: Rook: v1.9.12 ceph: 16.2.10 logs of kubelet: Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ]

"Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85-
FYI. this issue is fixed in 1.24.14, 1.25.10, 1.26.5, and has been fixed in v1.27.0
Hello , can you have a look at this reconciler run's operationExecutor fails to umount volume due to missing vol- It appears to be Kube v1.22 not .25 to boot is this the same issue?
I think they are the same issue and I have left some comments in
Issue reoccurs in kubelet version


Question: type Mount struct { // Path of the mount within the container. ContainerPath string `protobuf:"bytes,1,opt,name=container_path,json=containerPath,proto3" json:"container_path,omitempty"` <del> // Path of the mount on the host. </del> <ins> // Path of the mount on the host. If the hostPath doesn't exist, then runtimes // should report error. If the hostpath is a symbolic link, runtimes should // follow the symlink and mount the real destination to container. </ins> HostPath string `protobuf:"bytes,2,opt,name=host_path,json=hostPath,proto3" json:"host_path,omitempty"` // If set, the mount is read-only. Readonly bool `protobuf:"varint,3,opt,name=readonly,proto3" json:"readonly,omitempty"`

Currently, Docker create the host directory if the host path does not exist. For host path, Kubelet doesn't check the host directory existence, and pass the mount to docker directly. Given so, we should: 1) Either clearly define the expectation in CRI; 2) Or create the host directory ourselves when it doesn't exist. /cc
I think hostPath is expected to be already exist and kubelet should create the host directory if it is not exist.
Hmm, another problem is how to known whether hostPath is (expected) file or directory? Since it doesn't exist, we can only assume it is a directory?
There is a new field in the hostpath spec, that may help:
Another thing we may want to clearly define is: If when the host path is a symlink, what is the expected behavior. I think we assume container runtime should follow the symlink and mount the real content, but it's not clearly defined anywhere.
can you clarify, when you say "host path", are you referring specifically to hostPath volumes, or to any volume plugin's host-side path that gets passed into CRI?
I believe it's .
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: <ins> influxdb-go =========== </ins>

Due to changes in field selectors (see ) heapster was almost certainly busted, but continues to pass. This seems incorrect.


Question: }, }, }, <ins> { ObjectMeta: metav1.ObjectMeta{ Namespace: "node_info_cache_test", Name:      "test-3", UID:       types.UID("test-3"), }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceCPU: resource.MustParse("200m"), }, }, Ports: []v1.ContainerPort{ { HostIP:   "127.0.0.1", HostPort: 8080, Protocol: "TCP", }, }, }, }, InitContainers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("500m"), v1.ResourceMemory: resource.MustParse("200Mi"), }, }, }, }, NodeName: nodeName, Overhead: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("500m"), v1.ResourceMemory: resource.MustParse("500"), }, }, }, </ins> } expected := &NodeInfo{ node: &v1.Node{

The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!


Question: } // Refresh expired token. <del> if c.token.Expired() { </del> <ins> if !c.token.Valid() { </ins> token, service, err := connect() if err != nil { return nil, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted


Question: "//vendor/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/runtime:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/apimachinery/pkg/watch:go_default_library", "//vendor/k8s.io/client-go/pkg/api/v1:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-20T20:37:18Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Local: El Capitan, Remote: GCI Kernel (e.g. ): Local: Darwin ... 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep 1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASEX8664 x86_64 Install tools: gcloud Others: n/a What happened: Provisioned 3 clusters each in different regions (us-central1, asia-east1, europe-west1). Each cluster is in 2 zones (w/ gcloud --zone... --additional-zones...) Provisioned Federation API Server and Controller (1.4.4) based on Verified clusters were ready. Provisioned federated service & deployed backends into each of the cluster. Verified the federated services were created, and all of the regional LB were created, and verified all the LB IPs were working w/ direct requests. DNS provider updated DNS w/ most of the service/zone CNAME records. But A records did not complete. Further investigation for the log points to multiple entries in rrdata w/ the same IP due to the multi-zone deployment: I1029 19:24:24. 1 ] Have recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180.

Need recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180I1029 19:24:24. 1 ] Existing recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180is not equivalent to needed recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180, removing existing and adding needed. After creating the clusters w/ only one zone, the DNS provider worked properly. What you expected to happen: Cloud DNS Provider should update the DNS record properly w/ multi-zone clusters. How to reproduce it (as minimally and precisely as possible): 3 x multi-zone clusters Federation API Server and Controller clusters in to federation federated service and backends Anything else do we need to know: Using GKE and Cloud DNS
/cc
Also hit this. Any workarounds?
Sorry, no direct workarounds available yet. You could convert your multi-zone clusters into multiple single-zone clusters, and federate all of those.
FYI, although I'm sure you're aware - also affects AWS multi AZ clusters (1.6.2)
This seems bad. I'll try to take a look into it.
Fixed by


Question: } } } <ins> func TestIsCurrentInstance(t *testing.T) { cloud := &Cloud{ Config: Config{ VMType: vmTypeVMSS, }, } testcases := []struct { nodeName       string metadataVMName string expected       bool expectError    bool }{ { nodeName:       "node1", metadataVMName: "node1", expected:       true, }, { nodeName:       "node1", metadataVMName: "node2", expected:       false, }, { nodeName:       "vmss000001", metadataVMName: "vmss_1", expected:       true, }, { nodeName:       "vmss_2", metadataVMName: "vmss000000", expected:       false, }, { nodeName:       "vmss123456", metadataVMName: "vmss_$123", expected:       false, expectError:    true, }, } for i, test := range testcases { real, err := cloud.isCurrentInstance(types.NodeName(test.nodeName), test.metadataVMName) if test.expectError { if err == nil { t.Errorf("Test[%d] unexpected nil err", i) } } if real != test.expected { t.Errorf("Test[%d] unexpected isCurrentInstance result %v != %v", i, real, test.expected) } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider Instances interfaces don't work expected on Windows containers. This is because Windows containers don't support HostNetwork mode, while Azure cloud provider depends on "os.Hostname()" to check whether the VMName from IMDS is same with NodeName: To fix it, the dependency on Hostname should be removed. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /priority critical-urgent /area provider/azure /milestone v1.18


Question: tpNeedsUpdate = true } if supportsNodesHealthCheck { <del> hcToCreate = makeHttpHealthCheck(makeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort()) </del> <ins> hcToCreate = makeHttpHealthCheck(MakeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort()) </ins> } } // Now we get to some slightly more interesting logic.

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: "limitrange-updater":               {Name: "limitrange-updater"}, "limitrange-patcher":               {Name: "limitrange-patcher"}, "user-with-no-permissions":         {Name: "user-with-no-permissions"}, <del> })) </del> <ins> }))) </ins> controlPlaneConfig.GenericConfig.OpenAPIConfig = framework.DefaultOpenAPIConfig() _, s, closeFn := framework.RunAnAPIServer(controlPlaneConfig) defer closeFn()

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users