Question: return fmt.Errorf("%v", strings.Join(errMsg, "; ")) } <del> // NodeRunningOutdatedKubelet returns true if the kubeletVersion reported // in the nodeInfo of the given node is "outdated", meaning < 1.2.0. // Older versions were inflexible and modifying pod.Status directly through // the apiserver would result in unexpected outcomes. func NodeRunningOutdatedKubelet(node *v1.Node) bool { v, err := utilversion.ParseSemantic(node.Status.NodeInfo.KubeletVersion) if err != nil { glog.Errorf("couldn't parse version %q of node %v", node.Status.NodeInfo.KubeletVersion, err) return true } if v.LessThan(podStatusReconciliationVersion) { glog.Infof("Node %v running kubelet at (%v) which is less than the minimum version that allows nodecontroller to mark pods NotReady (%v).", node.Name, v, podStatusReconciliationVersion) return true } return false } </del> // NodeExistsInCloudProvider returns true if the node exists in the // cloud provider. func NodeExistsInCloudProvider(cloud cloudprovider.Interface, nodeName types.NodeName) (bool, error) {

As discussed on slack with we might want to clean the node controller from code that ensures backward compatibility for version 1.4 and earlier ones. /sig node
/assign
- how do we plan to devide this one ? The following order for merge should be fine I think: the tests (we can split also test case deletion if it's too long to review): logic itself + kubeletVersion on the according tests: change how we create nodecontroller: clean consts and BUILD:
Yeah, sounds good.
I'll create the PRs accordingly and assign them to you


Question: return err } opts.Config = config <del> opts.AttachFunc = attach.DefaultAttachFunc </del> <ins> opts.AttachFunc = attachFunc </ins> <del> clientset, err := kubernetes.NewForConfig(config) </del> <ins> clientset, err := f.KubernetesClientSet() </ins> if err != nil { return err }

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: } func (r *SimpleProxyReactor) Handles(action Action) bool { <del> resourceCovers := r.Resource == "*" || r.Resource == action.GetResource().Resource if !resourceCovers { return false } return true </del> <ins> return resourceCovers(r.Resource, action) </ins> } func (r *SimpleProxyReactor) React(action Action) (bool, restclient.ResponseWrapper, error) { return r.Reaction(action) } <ins> func resourceCovers(resource string, action Action) bool { if resource == "*" { return true } if resource == action.GetResource().Resource { return true } if index := strings.Index(resource, "/"); index != -1 && resource[:index] == action.GetResource().Resource && resource[index+1:] == action.GetSubresource() { return true } return false } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Can't match "serviceaccounts/token", can only match "serviceaccounts" Can match "serviceaccounts/token"
/kind feature
/sig testing
/assign
/triage accepted


Question: kc.ClusterDomain, net.IP(kc.ClusterDNS)) <ins> if err != nil { return nil, err } </ins> k.BirthCry() go k.GarbageCollectLoop() go kubelet.MonitorCAdvisor(k, kc.CAdvisorPort) kubelet.InitHealthChecking(k) <del> return k </del> <ins> return k, nil </ins> }

Things like SyncFrequency should have a lower-bound of acceptable, and that really should be 0ns.
The main point of this is now fixed, though we could be better about input checking all over


Question: REGISTRY ?= gcr.io/kubernetes-e2e-test-images GOARM=7 QEMUVERSION=v2.9.1 <del> GOLANG_VERSION=1.11.2 </del> <ins> GOLANG_VERSION=1.11.3 </ins> export ifndef WHAT

The go team has released 1.11.3 with three security fixes. From their e-mail announcement: We should kick off upgrades of both the kubernetes build system, and our CI, as soon as possible. Do you have an ETA for the rules_go bump that will contain 1.11.3 toolchain support? cc:
I'm preparing releases that use Go 1.11.3 by default. Those will be on the rules_go 0.14, 0.15, and 0.16 branches. I expect to tag them tonight as soon as they get through CI.
Fantastic! Thank you
rules_go 0.14.7, 0.15.9, and 0.16.4 are now tagged and ready to go.
related commits and issue:
Reopening to track other branches, as well as the CI bump
go1.11.3 and go1.10.6 introduced a regression, and so go1.11.4 and go1.10.7 were released Friday afternoon with a fix. We'll need to bump everything again on master, and should hold off on cherry-picks for now.
This looks fixed by Is there anything else that needs to be done as well?
Still need to merge for k8s 1.13.
Closing as all branches are now up to date.


Question: const ( DefaultKubeletPodsDirName                = "pods" DefaultKubeletVolumesDirName             = "volumes" <ins> DefaultKubeletVolumeSubpathsDirName      = "volume-subpaths" </ins> DefaultKubeletVolumeDevicesDirName       = "volumeDevices" DefaultKubeletPluginsDirName             = "plugins" DefaultKubeletPluginsRegistrationDirName = "plugins_registry"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Network for all nodes of kubernetes cluster (29 hosts) was turned off for a day. Once the network was brought back, we've scaled down a statefulset that mounts a from local volume which on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. In fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. The issue would continue until volume-subpath was manually unmounted (with command). Ended up removing 3rd of all of our data (we have 3 drives per host and one was wiped) for the whole cluster. Equivalent definition. Note that host path of local volume has to be on root drive: Kubelet logs (filtered out logs from other pods) during the issue: What you expected to happen: Expected no data loss for peristent local volumes. How to reproduce it (as minimally and precisely as possible): Unfortunately was not able to reproduce. But this bug hit 26 out of 29 nodes simultaneously. Need some way to create a case where a pod becomes orphan and "volumes" folder gets somehow removed (similar to issue ). Then there is a code path that can lead to data loss: We are in orphan pod cleanup and directory is missing (not sure why it's printed twice): No error returned and empty volumes list, so we go into this case which proceeds removing pod directory and along with it even thought they might still be mounted (not sure why): checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing (this is a bug IMHO). Anything else we need to know?: k8s version upgrades went from 1.9.6 -1.11.1 -1.12.1 -1.12.2 -1.13.0 without restarting the pods.

Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OVH, baremetal servers OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Using Ansible tasks that downloads kubelet from and all other required tools. Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thank you for digging into the code. I think there are a number of things that can be done here: why returned no volumes. why base volume was successfully unmounted. Unmounting the volume should have failed because the subpath bind mount would have kept a reference on it. to also look for the volume-subpaths directory to detect bind mounts.
I am working on a fix for 3 and 4
, do you mind to share us the full kubelet log? You can email me jinxu at Thanks!
why podVolumesExist returned no volumes. From the available log, we can see that Unmount volume and Unmount device succeeded. That's why podVolumeExist will return no volume.
/priority critical-urgent
That's unexpected. The subpath bind mount should have prevented the local volume bind mount from successfully unmounting.
Maybe the fact that the local volume mount is itself a bind mount could be the reason
about to why podVolumesExist returned no volumes. I think we do have an issue here podVolumesExist checks pod volume paths and whether they are mounted. Suppose it should stop cleaning up if there are paths still mounted. However, it only checks normal pod volumes /pod-id/volumes, it does not check subPath /pod-id/volume-subpath directories. We should fix this.
I'm adding a check for volume-subpath directories to address 3)
Jing found that the subpath volume directory uses the outer volume spec name, which is wrong. We made a fix for this in 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16, which changed the on-disk directories to use the inner volume spec name.

had an that in-place node updates were not supported and you would have to drain your pods first. This explains why the volume cleanup proceeded without waiting for the subpath mount cleanup.
We haven't run a version below 1.9.6 on this node.
I've sent the entire kubelet log to your email
to clarify, 1.9.6 uses the old on-disk directory name, so upgrading to anything at least 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16 requires pod draining first.
Right, I remember stumbling upon when debugging this. However, the release notes say: Shouldn't it specify exact version range where the OuterVolumeSpecName was used (1.9.4-1.9.6)? Unless I'm misreading release notes. I think it would be useful to clarify that.
ah you're right. The release notes should say from at least versions 1.7.14, 1.8.9, and 1.9.4
(will update it)
regardless, I still intend to add more safety checks to this orphaned pod cleanup function
From the log provided by the customer, I think we can confirm the reason of this issue is what mentioned By adding the safety checks during pod cleanup will help avoid this issue.


Question: tryConn.Close() connParams := grpc.ConnectParams{ <del> Backoff: backoff.Config{ BaseDelay: baseBackoffDelay, MaxDelay:  maxBackoffDelay, }, </del> <ins> Backoff: backoff.DefaultConfig, </ins> } <ins> connParams.Backoff.BaseDelay = baseBackoffDelay connParams.Backoff.MaxDelay = maxBackoffDelay </ins> gopts := []grpc.DialOption{ grpc.WithInsecure(), grpc.WithContextDialer(dialer.ContextDialer),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I restart containerd.service, kubelet can not connect to it again and cause 100% cpu usage. there are many logs like this: I can show container info via crictl at the meantime and restart kubelet.service can resolve the problem. What you expected to happen: kubelet can reconnect to containerd after containerd restart. How to reproduce it (as minimally and precisely as possible): cpu usage and syslog. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: armbian(debian on arm64) OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Reproduce: can reproduce the issue by restarting when is the container runtime another setup where was used used as container runtime, didn't experience the issue by restarting , but restarted the underlying , same issue shown up again and had to restart to resolve the issue. Will continue with the investigation, might need to reconnect to after the socket is broken.
To me it seems there are a couple of problems here. 1. kubelet does not reconnect to the containerd socket. 2. The reconnect is agressive and consumes a lot if cpu. I don't know the cause for 1, but for 2 this might be an indication that the retry interval should be relaxed.
Spent a few hours on the investigation, it's likely to be some internal states being messed up in .
TL;DR: It should be related to gRPC.

Hacked the vendored gRPC mod and changed its to something like this: Then CPU only spike for a few seconds after restart then comes back to normal instead of failing the all the time. kubelet passes a customized dialer to the gRPC client. I logged the address during kubelet startup: As you can see, the address of the is at . Before restarts, I can see from log that every time, the gRPC function is using the customized dialer that kubelet passed: However, once got restarted, the address of the dialer is changed: And that function at keeps failing. As I forced to use the original customized dialer, the issue is gone. Without too much knowledge on gRPC, my blind guess is as is passed by reference to gRPC client, some component could modify it during the process lifetime and the change got persisted. Anyway, I'll check with gRPC team on this issue and see if there's already a fix on their side. If so then I can port the fix here.
/cc
Hi, are there any updates on this. I am experiencing what appears to be the same issue. If I restart containerd while the kubelet is running, my kubelet logs get flooded with: Restarting the kubelet appears to be the only way to re-establish the connection with containerd. Version information is:
Hi thanks for the info, I'm still working on the root cause. As I mentioned above, it's something related to gRPC. I collected the object info, this is the object before containerd restart: ! And this is the object after containerd restart: ! There's one thing made me concern is, the connection state is still 2 (i.e. , see ) after the failure. I'm not sure if this is related to the cause but anyway it still should be something related to gRPC. Btw I've followed their to use the latest gRPC client without customized dialer and still see this issue, will still work on the root cause.
We're seeing this now when we update our nodes to 1.19 where our Kata Containers install does a . Our current workaround is to also do , but would be good to get this fixed.
Is this MR relative to the problem?

No I don't think so, I've cherry-picked that change and tried in my environment weeks back, and the issue is still there. I only get around the issue by modifying the gRPC code as I mentioned in the above, I'll continue to check which component modified the function in after thanksgiving.
For the sake of priority. This hit us hard these days too, as security updates for container.d triggered a restart and a ton of nodes flooded the central log management. ;) Thanks for working hard on a fix.
Same here. Two systems were affected by automatic container.d upgrades & restarts. But now I wonder if - generally - "unattended-upgrades" (Ubuntu 18.04) is a wise thing to do? Or should the OS for kubernetes deployments always be updated manually & controlled? I can not find best practices for this in the documentation.
After the recent CVE-2020-15257many users will upgrade and restart containerd. It would be a big problem.
I see, will prioritize on the root cause and fix. At the mean time, you need to restart after you restart .
Had another round of debugging, and made one step towards the root cause. I following snipple to the as I mentioned in previous : When I reproduced the error, I got following information for the dialer function: That leads to cadvisor, which is the only component that uses the in the repo: So I'm convinced that this should be related to and will update the thread once I've made further progress.
EOD updates: issue is indeed caused by cadvisor, reproduced the issue in this . confusing timeout in log should be . root cause might be a bug for using context that already exceeded deadline in cadvisor. To validate that, I need to explore all methods of this : Will work on the fix on cadvisor for the rest of the week.
Fix was made in Copying from to ensure we can capture it on the issue here: I've cut new cAdvisor releases (v0.37.3 and v0.38.6) with this fix cherrypicked. We should get the cAdvisor fix back into kubernetes.

We'll need 3 PRs to k/k with following updates: [x] k/k master to update cAdvisor to v0.38.6 -- [x] k/k release-1.20 to update cAdvisor to v0.38.6 -- [x] k/k release-1.19 to update cAdvisor to v0.37.3 --
Thanks for fixing this one, definitely a tricky one to track down.
Thanks for all the patience and team efforts on getting this issue fixed, also credits to on the guides and discussions, won't be able to get it done without your helps!
The fix was applied on master as well as cherrypicked into and releases.
I remember you told me, but I forgot. Does issue exist in 1.18 and 1.17?
I don't believe 1.18 or 1.17 should be affected. I think this issue was introduced with which commit was included in cAdvisor v0.37 release. cAdvisor v0.37 was vendored into k8s 1.19.


Question: } } <del> // runCommand runs the cmd and returns the combined stdout and stderr, or an // error if the command failed. func runCommand(cmd ...string) (string, error) { output, err := exec.Command(cmd[0], cmd[1:]...).CombinedOutput() if err != nil { return "", fmt.Errorf("failed to run %q: %s (%s)", strings.Join(cmd, " "), err, output) } return string(output), nil } </del> // getCRIClient connects CRI and returns CRI runtime service clients and image service client. func getCRIClient() (internalapi.RuntimeService, internalapi.ImageManagerService, error) { // connection timeout for CRI service connection

All the crio serial jobs (presubmits and periodics) are not displaying the test grid results and timing out. periodics: node-kubelet-serial-crio presubmits: pull-kubernetes-node-kubelet-serial-crio-cgroupv1 pull-kubernetes-node-kubelet-serial-crio-cgroupv2 At least 6 months If you look into the artifacts the jobs are running but there seems to be hitting a timeout in the e2e_node tests. This causes the grid to not post the results so it marks the job as failed. We tried increasing the timeout to 6 hours from 4 hours but the timeout still occurs. Something is not right and we could use some help on figuring out why this can be happening. is one example that works without issue. /sig node testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Looks like our recent PR didn't fix this.
Looking into the serial logs, it looks like these tests are hitting OOM. [ 3276.] Out of memory: Killed process (dd) total-vm:10490032kB, anon-rss:3218048kB, file-rss:80kB, shmem-rss:0kB, UID:0 pgtables:6352kB oomscoreadj:1000 [ 3276.] Out of memory: Killed process (dd) total-vm:10490032kB, anon-rss:3218048kB, file-rss:80kB, shmem-rss:0kB, UID:0 pgtables:6352kB oomscoreadj:1000
/reopen Memory limits did not help..
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Bumping machine size didn't fix this.
/cc :eyes:
dd is run in a couple of tests as far as I can see: running dd to simulate disk pressure, is similar. So, I'd start with The only Serial test I can find is this one: I've removed label from it to see it helps the test, here is a test PR: Let's see how it goes.
This looks quite dangerous and explains why bigger instances and limits didn't help: It tries to allocate 100Gb of memory.
Successful run has 'Out of memory: Killed process' as well, e.g.
It looks like OOMKiller message is expected and doesn't impact the issue. It's produced by the test case mentioned above. Without the test case the job is still getting timeouted and doesn't produce artifacts. Here is an example of failed and timeouted job run without the test case:
I don't think there is something wrong with long running tests. There are a lot of slow tests in the job and if infra is loaded with other tasks all of the tests would run slower. Here is a list of test durations for successful and failed job runs. I don't see anything that can't be explained by the busyness of the infra. Please correct me if I'm wrong here. successful run ()failed run (): One strange thing though is that the total time for the failed test(.577 sec / 60 = 365.2596m) is less than the job timeout (420m). Any ideas why the job was timed out?
Without slow tests finished in around 1h 25m and produced valid artifacts.
Without slow tests and with OOMKiller test the job still works ok:
With all tests the issue persisted despite the 10h timeout: I'll start bisecting. Hopefully will spot the test that causes everything to slow down.
One thing I noticed is that these tests are running two times. You can search for the start tests banner and see that we restart if the test failed.
One of the failures in the failed link was because kubelet restart took more time than ImageMaximumGCAge of 1 minute, is this expected ?

Attempt to fix this
Looking into what tests take the longest, I notice that the *Manager jobs are all pretty beefy. WDYT of moving these to a separate serial job?
An interesting datapoint I found between containerd serial and crio serial: in containerd: On crio, it takes 974 seconds.
Running this test on gcp with crio I don't see why it is taking up that long. I think that there is something else going on in this test. In the tests that were passing this test only take 14 seconds also.
With all but last 3 slow tests excluded the job still run more than 8h: Remaining slow tests: Resource-usage, Restart and SystemNodeCriticalPod In contrast the job with all slow tests excluded took only 1h25m: I'm not sure if the reason of slowness is in one of those 3 tests or it's something else. I'll remove them and run the job again.
I don't think they're running 2 times. The build log is duplicated if testrun fails, I guess. Look at the duration time for the same test case. They're exactly the same with a high precision, which wouldn't be possible for two different test runs.
Job run with only and slow tests enabled took more than 5 hours and didn't produce enough artifacts:
Successful run is executed on , failed is executed on . Could it be something related with this datapoint?
job is a current suspect. Test run with enabled and takes 1h 46 min and looks good: I'll try to enable more slow jobs with excluded job.
Job run with enabled resourceusage, systemnodecritical, log rotation, density and eviction tests takes 1h51m and looks good: So far so good. Side note: I'm not sure I understand why logrotation, density and eviction tests marked slow as they altogether take around 5 minutes.
Job run with all tests except of takes 2h and still looks good: I'll try to investigate test.
Job run with all tests except of and takes 2h 4m. and still looks good:
Can you point out to those runs, please? In my testing both pull jobs use n1-standard-4 and results don't differ.

I was referring to the job shared in the description
Job run with all tests except of takes 2h 9m and looks good: So, test case seems to slow down everything.
Nice! So we can just filter out that job then? Not sure if we have a slow lane for crio/containerd?
checking the code it seems it is performing an Ubuntu specific task. Is this applicable/needed also for Fedora?
is a test case in the test: We can either remove/comment out it or investigate why it slows down everything.
It looks like it restarts dbus and systemd, which theoretically should work on any linux with systemd and dbus. However, that comment could mean that it was never tested on anything except Ubuntu.
The Dbus test case was by Can you help us to investigate why test case slows everything down on Fedora/CoreOS ?
The test case was by and tested only with as crio pull jobs didn't exist at that time. However, it should be failing/slowing down ci jobs if they exist.
I opened up I talked with a few people at Red hat and it was suggested that this test is bad. Going to try just skipping it on fedora. Restarting dbus is not recommended
I asked as he has been working on gracefulshutdown and asked about a few of these problems on RHEL like OS. I think skipping should be fine for this.
Fixed remaining failed tests. Both jobs are green now:
/assign
Nice job


Question: } } <ins> func TestIsValidGroupId(t *testing.T) { goodValues := []int64{0, 1, 1000, 65535, 2147483647} for _, val := range goodValues { if !IsValidGroupId(val) { t.Errorf("expected true for '%d'", val) } } badValues := []int64{-1, -1003, 2147483648, 4147483647} for _, val := range badValues { if IsValidGroupId(val) { t.Errorf("expected false for '%d'", val) } } } func TestIsValidUserId(t *testing.T) { goodValues := []int64{0, 1, 1000, 65535, 2147483647} for _, val := range goodValues { if !IsValidUserId(val) { t.Errorf("expected true for '%d'", val) } } badValues := []int64{-1, -1003, 2147483648, 4147483647} for _, val := range badValues { if IsValidUserId(val) { t.Errorf("expected false for '%d'", val) } } } </ins> func TestIsValidPortName(t *testing.T) { goodValues := []string{"telnet", "re-mail-ck", "pop3", "a", "a-1", "1-a", "a-1-b-2-c", "1-a-2-b-3"} for _, val := range goodValues {

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: <ins> # Maintainers Eric Paris <eparis@redhat.com> </ins>

This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.22, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /assign /sig release /milestone v1.22
GitHub didn't allow me to assign the following users: simcard0000. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.19, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /sig release /milestone v1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
1.19 or 1.22 ?
/assign
For the 1.21 example in , should be changed to
Please add the following to the Known Issues section of the 1.22 Release Notes: CC
Can you also mention the CSIMigrationVSphere feature gate in the note as well?
what should I mention about the CSIMigrationVSphere feature gate? If someone uses vSphere CSI Driver, he/she should not upgrade to v1.22 whether CSIMigrationVSphere is on or off. If someone does not use vSphere CSI Driver, then we also don't need to worry about this feature gate.
If the feature gate is enabled, they should also hold off on upgrading to 1.22
The main concern as far as k8s release is - it is intree to CSI migration of vSphere volumes that is broken in 1.22. So I think our release note should say something like:


Question: return err } <ins> func updateNode(cs clientset.Interface, node *v1.Node) (*v1.Node, error) { return cs.CoreV1().Nodes().Update(context.TODO(), node, metav1.UpdateOptions{}) } </ins> func createNode(cs clientset.Interface, node *v1.Node) (*v1.Node, error) { return cs.CoreV1().Nodes().Create(context.TODO(), node, metav1.CreateOptions{}) }

We have an e2e test that involves cluster autoscaler; the test creates two pods back to back which forces CA to scale up the cluster to create two nodes also back to back one for each pod. The two nodes are not created at the same time, there is some delay between them. The problem is that the second pod never gets scheduled. We see in the logs a single scheduling attempt for the second pod that attempted to schedule it on the first node and failed because it didn't fit the node's remaining cpu cores. We don't see any attempts after that for the pod and the test times out waiting for that pod to schedule. What changed in the scheduler: Pre 1.22, on most update/add/delete events to pods/nodes/pvc/storageClass resources, we used to move all unschedulable pods back to the queue. Also every we did force move all unschedulable pods back to the queue indiscriminately. This behavior is not efficient, wasted scheduling cycles and since we re-queue based on last scheduling attempt timestamp, it missed up badly with the best effort FIFO behavior we intend to have for pod scheduling.Since 1.22 a couple of things changed such that unschedulable pods are not placed back into the queue on those events indiscriminately; on each such event, we iterate over unschedulable pods and we place them back into the queue if they pass two checks: the first one happens only on node update/add events, it is carried by the function ; it basically runs the kubelet admission logic (node resource, affinity, taints/tolerations and host port checks) the second check runs on all events (including the node ones); this one is a little more complicated and fine grained: for each unschedulable pod we remember which plugins last failed to fit the pod; at the same time plugins register which events could make an unschedulable pod schedulable (e.g. nodeResources filter plugin registers pod delete and node add/update events). A pod is put back into the queue only if the event matches any of the ones registered by the plugins that last failed the pod.

In 1.24 we increased the 60s to (our north star to completely remove this) What went wrong: My theory is that we incorrectly determined that the pod was not eligible to be put back into the queue on events related to the new node; and that the 5min timeout was too long to force moving it back to the queue (I am guessing the whole test timed out), while the 60s was basically masking the problem for 1.22 and 1.23. So what checks are incorrectly failing to move the pod? Here is what happened: Before adding the new node, the pod failed the nodeResource plugin, which only registers for New node was , scheduler receives an add event. The pod was not moved back to the queue because it failed the check on NotReady taint. When the new node became ready, the scheduler receives a node update event, but since it is a taint update event, it didn't match the events that the nodeResources plugin registered for. How to fix it: Since we have the ; pods that previously failed because of NodeResources, NodePorts, TaintsAndTolerations and NodeAffinity (the predicates we already ran in ) should always be placed back into the queue; basically those plugins should register for all node update events. /sig scheduling /cc The second pod to schedule. See above. No response 24 GKE <details</details<details</details<details</details<details</details
/triage accepted


Question: return fmt.Errorf("deployment %q failed to create new replica set", deploymentName) } if err != nil { <del> return fmt.Errorf("error waiting for deployment %q (got %s / %s) and new replica set %q (got %s / %s) revision and image to match expectation (expected %s / %s): %v", deploymentName, deployment.Annotations[deploymentutil.RevisionAnnotation], deployment.Spec.Template.Spec.Containers[0].Image, newRS.Name, newRS.Annotations[deploymentutil.RevisionAnnotation], newRS.Spec.Template.Spec.Containers[0].Image, revision, image, err) </del> <ins> if deployment == nil { return errors.Wrapf(err, "error creating new replica set for deployment %q ", deploymentName) } deploymentImage := "" if len(deployment.Spec.Template.Spec.Containers) > 0 { deploymentImage = deployment.Spec.Template.Spec.Containers[0].Image } newRSImage := "" if len(newRS.Spec.Template.Spec.Containers) > 0 { newRSImage = newRS.Spec.Template.Spec.Containers[0].Image } return fmt.Errorf("error waiting for deployment %q (got %s / %s) and new replica set %q (got %s / %s) revision and image to match expectation (expected %s / %s): %v", deploymentName, deployment.Annotations[deploymentutil.RevisionAnnotation], deploymentImage, newRS.Name, newRS.Annotations[deploymentutil.RevisionAnnotation], newRSImage, revision, image, err) </ins> } return nil }

Found in local testing. Looks like it's coming from:
/area conformance
/sig testing


Question: ginkgo.It("should never report success for a pending container", func() { ginkgo.By("creating pods that should always exit 1 and terminating the pod after a random delay") <del> var reBug88766 = regexp.MustCompile(`rootfs_linux.*kubernetes.io~secret.*no such file or directory`) </del> <ins> var reBug88766 = regexp.MustCompile(`rootfs_linux.*kubernetes.io~(secret|projected).*no such file or directory`) </ins> var ( lock sync.Mutex

ci-kubernetes-e2e-gce-scale-correctness [sig-node] Pods Extended Pod Container Status should never report success for a pending container 2021-02-19 https://k8s- ! Feb 19 12:22:34.933: 1 errors: pod pod-submit-status-0-11 on node gce-scale-cluster-minion-group-2-cczj container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2021-02-19 12:21:43 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-96dc-4682-af6a-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-96dc-4682-af6a- no such file or directory""": unknown /sig scalability /sig node It failed 6/10 times in recent runs /priority critical-urgent /cc /cc
/triage accepted /cc
I do not see any recent changes in test setup, apart from test renames.
Test is already ignoring: Looks very similar, so probably regex should be extended. Original issues and PRs: /assign
could you triage?
We meet tomorrow, I'll make sure this is on our board/agenda.
Injected service account tokens changed from secret-based to projected-token based by default in , which merged 2/18 That coincides with this flake starting: as points out, the test already tolerated secret volume setup errors, I guess because those error messages are expected to occur spuriously in rapidly restarting containers? If it was acceptable for secret volume errors to be encountered in this test before, I agree the regex should be expanded to accomodate projected volume tokens. opened to do this. Separately, is sig storage or sig node looking at


Question: nodeName := types.NodeName("instance") readOnly := false spec := createVolSpec(diskName, readOnly) <del> attachError := errors.New("Fake attach error") </del> detachError := errors.New("Fake detach error") diskCheckError := errors.New("Fake DiskIsAttached error") tests := []testcase{ // Successful Attach call { <del> name:   "Attach_Positive", attach: attachCall{diskName, nodeName, nil}, </del> <ins> name:           "Attach_Positive", diskIsAttached: diskIsAttachedCall{diskName, nodeName, false, diskCheckError}, attach:         attachCall{diskName, nodeName, nil}, </ins> test: func(testcase *testcase) (string, error) { attacher := newAttacher(testcase) return attacher.Attach(spec, nodeName)

Currently we don't check if disk is already attached : from what i remember it can cause some problems, but not sure, sorry for the mass tag
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage /assign


Question: import ( "path" "testing" <del> "time" </del> "github.com/stretchr/testify/require"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For now, UT in doesn't work because callback function will never be called, is rewritten in this function : Another thing, i'm wondering why we rewrite device health in function which in this file: What you expected to happen: UT assertion should be executed. How to reproduce it (as minimally and precisely as possible): Add some logs in callback function of , it will show nothing. Anything else we need to know?: I will send a PR to fix this. /cc WDYT? /sig node
/assign
GitHub didn't allow me to assign the following users: RenaudWasTaken. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Good catch


Question: APIVersion: "v1", FieldPath:  "spec.containers[3]", }, <del> Reason:  "Started", Message: "some verbose message: 1", Source:  v1.EventSource{Component: "eventTest"}, Count:   1, Type:    v1.EventTypeNormal, </del> <ins> Reason:              "Started", Message:             "some verbose message: 1", Source:              v1.EventSource{Component: "eventTest"}, ReportingController: "eventTest", Count:               1, Type:                v1.EventTypeNormal, </ins> }, expectLog:    `Event(v1.ObjectReference{Kind:"Pod", Namespace:"baz", Name:"foo", UID:"differentUid", APIVersion:"v1", ResourceVersion:"", FieldPath:"spec.containers[3]"}): type: 'Normal' reason: 'Started' some verbose message: 1`, expectUpdate: false,

From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


Question: DNS configuration MUST be configured in the Pod. release: v1.17 file: test/e2e/network/dns.go <ins> - testname: Scheduling, HostPort matching and HostIP and Protocol not-matching codename: '[sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]' description: Pods with the same HostPort value MUST be able to be scheduled to the same node if the HostIP or Protocol is different. This test is marked LinuxOnly since hostNetwork is not supported on Windows. release: v1.16, v1.20 file: test/e2e/network/hostport.go </ins> - testname: Ingress API codename: '[sig-network] Ingress API should support creating Ingress API operations [Conformance]'

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: We want to change ownership of a test from scheduler to network Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Once merges, we will exclude hostPort test on Windows. This is because of how the hostPort test is structured where we are relying on hostNetwork pod to test the communication between hostPort pods. This will not work on Windows as hostNetwork is not supported. I think it is better to have this test in sig-network and test the communication as scheduler should be responsible only for assigning a pod to node. This also ensures that hostPort on Windows is exercised properly. Anything else we need to know: More context
/sig scheduling /sig network cc
/area conformance /area test /sig windows since windows
/triage accepted


Question: FROM BASEIMAGE <ins> RUN echo CACHEBUST>/dev/null && clean-install  bash </ins> # The samba-common, cifs-utils, and nfs-common packages depend on <del> # ucf, which itself depends on /bin/bash existing. # It doesn't seem to actually need bash, however. RUN ln -s /bin/sh /bin/bash </del> <ins> # ucf, which itself depends on /bin/bash. RUN echo "dash dash/sh boolean false" | debconf-set-selections RUN DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash </ins> RUN echo CACHEBUST>/dev/null && clean-install  ca-certificates

the new hyperkube image has changed without notifying the bash environment now links to sh, there is only one thing worst than an error, and it is a SILENT ERROR this means that anyone that execute kubectl inside the hyperkube and leverage from a bash environment now get the weirdest exceptions , in my case troubles where caused by flexvolume scripted in bash which stopped working without reason and with weird errors logged ( but also there are some other cluster-images based on your hyperkube image, and if you link bash to sh it will spread headache around :) ) while for version 1.7.7 the result was different so , or remove bash, or repristinate bash, link bash-sh is a NO-NO
/assign /sig node


Question: if err != nil { return fmt.Errorf("fc: failed to unmount: %snError: %v", deviceMountPath, err) } <ins> // GetDeviceNameFromMount from above returns an empty string if deviceMountPath is not a mount point // There is no need to DetachDisk if this is the case (and DetachDisk will throw an error if we attempt) if devName == "" { return nil } </ins> unMounter := volumeSpecToUnmounter(detacher.mounter) err = detacher.manager.DetachDisk(*unMounter, devName) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Certain stress tests we run involve a large number of fibre channel persistent volumes shared amount multiple Kubernetes worker nodes. During the stress tests, any given one of these volumes may be bound by a PVC and mounted to a node, released, and bound by a different PVC and mounted to a different node. Given enough cycles (and generally the number is not that large), failures start to occur. The first symptom of a problem is a discrepancy between "volumesAttached" and "volumesInUse" for a node. Eventually, "volumesInUse" grows significantly larger than "volumesAttached", indicating that the attach/detach controller is forcefully detaching fibre channel volumes from a node even though the node's kubelet still believes those volumes are mounted. kubecontroller-manager logs like the following confirm that this is happening: W0911 15:57:53. 1 ] attacherDetacher.DetachVolume started for volume "array-00-ext-016" (UniqueName: "[]") on node "node4" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching Kubelet logs shed some light on what is happening: I0911 15:55:16. ] operationExecutor.UnmountDevice started for volume "array-00-xfs-050" (UniqueName: "[]") on node "node4" W0911 15:55:16. ] Warning: Unmount skipped because path does not exist: I0911 15:55:16. ] fc: DetachDisk devicePath: , dstPath: ., devices: [.] E0911 15:55:16. ] Operation for ""[]"" failed. No retries permitted until 2020-09-11 15:55:24. +0000 UTC m=+2146. (durationBeforeRetry 8s).

Error: "UnmountDevice failed for volume "array-00-xfs-050" (UniqueName: "[]") on node "node4" : fc: failed to detach disk: nError: fc: detachFCDisk failed. device: . err: fc detach disk: invalid device name: ." The kubelet's volume manager reconcile loop is repeatedly attempting to do an UnmountDevice on the fibre channel volume in question, but something in the code is broken. DetachFCDisk never receives the name of a device to detach. It throws and error, UnmountDevice fails, and the loop continues. After six minutes, the attach/detach controller gives up waiting for the kubelet to unmount the volume and allows it to be attached to a different node. The volume continues to appear in one node's "volumesInUse" list while it also appears in another node's "volumesAttached" and "volumesInUse" list. What you expected to happen: DetachFCDisk should receive a device name if a fibre channel volumes is being unmounted and needs to be detached. Otherwise, DetachFCDisk should not be called, and thus not throw an error that confuses the kubelet volume manager's reconcile loop. How to reproduce it (as minimally and precisely as possible): This may be difficult to reproduce, as it only seems to occur in the circumstances described above (a large number of fibre channel volumes shared by many nodes undergoing high churn). However, we will propose a simple fix that we have tested in our environment. The fix appears to eliminate the issue. Environment: Kubernetes version (use ): 1.14.3 Cloud provider or hardware configuration: bare-metal Lenovo servers OS (e.g: ): CentOS Linux release 7.7.1908 Kernel (e.g. ): 3.10.0-1062.18.1.el7.x8664 Install tools: kubeadm
/sig storage
/assign ejweber


Question: framework.ExpectEqual(timeWaited < deadline, true, "waited %s, wanted less than %s", timeWaited, deadline) }) }) <del> ginkgo.Describe("with long upscale stabilization window", func() { ginkgo.It("should scale up only after the stabilization period", func() { ginkgo.By("setting up resource consumer and HPA") podCPURequest := 500 targetCPUUtilizationPercent := 25 usageForSingleReplica := 110 initPods := 4 initCPUUsageTotal := initPods * usageForSingleReplica upScaleStabilization := 3 * time.Minute rc := e2eautoscaling.NewDynamicResourceConsumer( "consumer", f.Namespace.Name, e2eautoscaling.KindDeployment, initPods, initCPUUsageTotal, 0, 0, int64(podCPURequest), 200, f.ClientSet, f.ScalesGetter, e2eautoscaling.Disable, e2eautoscaling.Idle, ) defer rc.CleanUp() hpa := e2eautoscaling.CreateCPUHorizontalPodAutoscalerWithBehavior( rc, int32(targetCPUUtilizationPercent), 1, 5, e2eautoscaling.HPABehaviorWithUpscaleStabilizationSeconds(int32(upScaleStabilization.Seconds())), ) defer e2eautoscaling.DeleteHPAWithBehavior(rc, hpa.Name) fullWindowOfNewUsage := 30 * time.Second windowWithOldUsagePasses := 30 * time.Second newPodMetricsDelay := 15 * time.Second metricsAvailableDelay := fullWindowOfNewUsage + windowWithOldUsagePasses + newPodMetricsDelay hpaReconciliationInterval := 15 * time.Second actuationDelay := 10 * time.Second maxHPAReactionTime := metricsAvailableDelay + hpaReconciliationInterval + actuationDelay maxConsumeCPUDelay := 30 * time.Second waitForReplicasPollInterval := 20 * time.Second maxResourceConsumerDelay := maxConsumeCPUDelay + waitForReplicasPollInterval waitBuffer := 1 * time.Minute // making sure HPA is ready, doing its job and already has a recommendation recorded // for stabilization logic before increasing the consumption ginkgo.By("triggering scale down to record a recommendation") rc.ConsumeCPU(2 * usageForSingleReplica) rc.WaitForReplicas(2, maxHPAReactionTime+maxResourceConsumerDelay+waitBuffer) ginkgo.By("triggering scale up by increasing consumption") rc.ConsumeCPU(5 * usageForSingleReplica) waitStart := time.Now() rc.WaitForReplicas(5, upScaleStabilization+maxHPAReactionTime+maxResourceConsumerDelay+waitBuffer) timeWaited := time.Now().Sub(waitStart) ginkgo.By("verifying time waited for a scale up") framework.Logf("time waited for scale up: %s", timeWaited) framework.ExpectEqual(timeWaited > upScaleStabilization, true, "waited %s, wanted more than %s", timeWaited, upScaleStabilization) deadline := upScaleStabilization + maxHPAReactionTime + maxResourceConsumerDelay framework.ExpectEqual(timeWaited < deadline, true, "waited %s, wanted less than %s", timeWaited, deadline) }) }) </del> })

gce-cos-autoscaling-hpa-cpu Kubernetes e2e suite.[sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with long upscale stabilization window should scale up only after the stabilization period Since it was in (first run at 2022-06-14 13:13 CEST failed, all subsequent runs failed too). https://k8s- Times out when waiting for scale down (), logs from (): I'm not sure why exactly, I have 2 ideas: It looks like the consumer briefly consumes 440 mCPU, which could result in a higher recommendation being generated HPA stabilises initial size of the consumer Increasing timeout should fix the issue No response_ /sig autoscaling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI
This was already reverted (and later the test got fixed and merged). /close
Closing this issue. <detailsIn response to : This was already reverted (and later the test got fixed and merged). Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: uniqueVolumeName = volumehelper.GetUniqueVolumeNameForNonAttachableVolume(volume.podName, plugin, volumeSpec) } <ins> if attachablePlugin != nil { if isNotMount, mountCheckErr := rc.mounter.IsLikelyNotMountPoint(volume.mountPath); mountCheckErr != nil { return nil, fmt.Errorf("Could not check whether the volume %q (spec.Name: %q) pod %q (UID: %q) is mounted with: %v", uniqueVolumeName, volumeSpec.Name(), volume.podName, pod.UID, mountCheckErr) } else if isNotMount { return nil, fmt.Errorf("Volume: %q is not mounted", uniqueVolumeName) } } </ins> volumeMounter, newMounterErr := plugin.NewMounter( volumeSpec, pod,

/kind bug What happened: Volumes used by pod are not getting mounted after rebooting a node. Effects Flexvolume/FC/ISCSI volume types. Pod goes into running state using local file system path. What you expected to happen Volume should be mounted and pod should not go into running state if the volume fails to attach. How to reproduce it (as minimally and precisely as possible): Create a RS with Flexvolume/FC/ISCSI. Reboot the node. Volume would not be mounted after reboot. Anything else we need to know?: Kubelet on a process restart avoids remounting the volume. The same code path is used on a node reboot and it avoids remounting the volume. GCE, AWS etc seems to be working by chance as the SPEC reconstruction fails when a volume is not mounted on reboot. Environment: Kubernetes version (use ): 1.6.9 Cloud provider or hardware configuration**: Baremetal OS (e.g. from /etc/os-release): Centos 7.2 Kernel (e.g. ): 3.10.0-327.el7.x86_64
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
/cc


Question: func (c *fakeDiscoveryClient) OpenAPIV3() openapi.Client { panic("implement me") } <ins> type fakeCachedDiscoveryClient struct { discovery.DiscoveryInterface freshHandler      func() bool invalidateHandler func() } var _ discovery.CachedDiscoveryInterface = &fakeCachedDiscoveryClient{} func (c *fakeCachedDiscoveryClient) Fresh() bool { if c.freshHandler != nil { return c.freshHandler() } return true } func (c *fakeCachedDiscoveryClient) Invalidate() { if c.invalidateHandler != nil { c.invalidateHandler() } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine.

We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package podtopologyspread import ( "context" "fmt" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/kubernetes/pkg/scheduler/algorithm/predicates" "k8s.io/kubernetes/pkg/scheduler/framework/plugins/migration" framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" "k8s.io/kubernetes/pkg/scheduler/nodeinfo" ) // PodTopologySpread is a plugin that ensures pod's topologySpreadConstraints is satisfied. type PodTopologySpread struct{} var _ framework.FilterPlugin = &PodTopologySpread{} // Name is the name of the plugin used in the plugin registry and configurations. const Name = "PodTopologySpread" // Name returns name of the plugin. It is used in logs, etc. func (pl *PodTopologySpread) Name() string { return Name } // Filter invoked at the filter extension point. func (pl *PodTopologySpread) Filter(ctx context.Context, cycleState *framework.CycleState, pod *v1.Pod, nodeInfo *nodeinfo.NodeInfo) *framework.Status { meta, ok := migration.PredicateMetadata(cycleState).(predicates.PredicateMetadata) if !ok { return migration.ErrorToFrameworkStatus(fmt.Errorf("%+v convert to predicates.PredicateMetadata error", cycleState)) } _, reasons, err := predicates.EvenPodsSpreadPredicate(pod, meta, nodeInfo) return migration.PredicateResultToFrameworkStatus(reasons, err) } // New initializes a new plugin and returns it. func New(_ *runtime.Unknown, _ framework.FrameworkHandle) (framework.Plugin, error) { return &PodTopologySpread{}, nil } </ins>

What would you like to be : Add a filter plugin that calls into EvenPodsSpread predicate, example Why is this needed: Part of /sig scheduling /help /priority important-soon
/assign
/remove-help
/retitle: [migration phase 1] EvenPodsSpread as filter plugin
/retitle [migration phase 1] EvenPodsSpread as filter plugin
/assign /unassign
May I take over this?
is going to take this over after the offline discussion, and he is familiar with EvenPodsSpread
IC, thanks a lot.


Question: } } <ins> func TestRunControllerWithWrongArgs(t *testing.T) { fakeClient := client.Fake{} name := "name" image := "foo/bar" replicas := 3 err := RunController(api.NewDefaultContext(), image, name, replicas, &fakeClient, "8080:", -1) if err == nil { t.Errorf("Unexpected non-error: %#v", fakeClient.Actions) } RunController(api.NewDefaultContext(), image, name, replicas, &fakeClient, "8080:80", -1) if len(fakeClient.Actions) != 1 || fakeClient.Actions[0].Action != "create-controller" { t.Errorf("Unexpected actions: %#v", fakeClient.Actions) } controller := fakeClient.Actions[0].Value.(*api.ReplicationController) if controller.ID != name || controller.DesiredState.Replicas != replicas || controller.DesiredState.PodTemplate.DesiredState.Manifest.Containers[0].Image != image { t.Errorf("Unexpected controller: %#v", controller) } } </ins> func TestRunControllerWithService(t *testing.T) { fakeClient := client.Fake{} name := "name"

Kubernetes will create replication controller and pods in case of port spec error, e.g. will create 'myNginx'. A following correct spec will fail:
/cc


Question: client clientset.Interface // a means to list all known scheduled pods. scheduledPodLister corelisters.PodLister <del> // a means to list all known scheduled pods and pods assumed to have been scheduled. podLister algorithm.PodLister </del> // a means to list all nodes nodeLister corelisters.NodeLister // a means to list all PersistentVolumes

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Consider use schedulerCache to replace fields in PluginFactoryArgs ref: /assign
/sig scheduling /priority backlog


Question: CMD_TARGETS="${CMD_TARGETS} ${KUBE_CONFORMANCE_IMAGE_TARGETS[*]}" fi <ins> # TODO(dims): Remove this when we get rid of hyperkube image CMD_TARGETS="${CMD_TARGETS} cmd/kubelet cmd/kubectl" </ins> kube::build::verify_prereqs kube::build::build_image kube::build::run_build_command make all WHAT="${CMD_TARGETS}" KUBE_BUILD_PLATFORMS="${KUBE_SERVER_PLATFORMS[*]}"

What happened: make quick-release-images fails on v1.17.0, works fine on v1.16.4 What you expected to happen: Should work on v1.17.0 How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: The immediate cause of the problem seems to be that the hyperkube image creation fails because it doesn't find the kubectl and kubelet binaries in , apparently because they weren't built.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: fs.StringSliceVar(&s.EtcdServersOverrides, "etcd-servers-overrides", s.EtcdServersOverrides, ""+ "Per-resource etcd servers overrides, comma separated. The individual override "+ <del> "format: group/resource#servers, where servers are URLs, semicolon separated.") </del> <ins> "format: group/resource#servers, where servers are URLs, semicolon separated. "+ "Note that this applies only to resources compiled into this server binary. ") </ins> fs.StringVar(&s.DefaultStorageMediaType, "storage-media-type", s.DefaultStorageMediaType, ""+ "The media type to use to store objects in storage. "+

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Comment on the scope of --etcd-servers-overrides option in the generated documentation Why is this needed: The documentation does not say if the --etcd-servers-overrides option for kube-apiserver does support CRDs.
/sig docs
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
The doc could be clarified, but the following suggest the answer is no support for CRs unless recently:
Yes, indeed I think it should be at least clarified in the docs.


Question: // That will typically happen with init-containers in Exited state. Docker still knows about them but the HCS does not. // As we don't want to block stats retrieval for other containers, we only log errors. if !hcsshim.IsNotExist(err) && !hcsshim.IsAlreadyStopped(err) { <del> klog.Errorf("Error opening container (stats will be missing) '%s': %v", containerID, err) </del> <ins> klog.V(4).Infof("Error opening container (stats will be missing) '%s': %v", containerID, err) </ins> } return nil, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When reviewing Windows kubelet logs noticed errors like: What you expected to happen: The stats should be returned How to reproduce it (as minimally and precisely as possible): This occurs when the pods has just started or during termination. To simulate the issue: Then deploy pod: Anything else we need to know?: The container is in a state but is has not started running so the call to get the stats fails: This is handled in other places like moby: and Environment: Kubernetes version (use ): 1.18+ Cloud provider or hardware configuration: aks-engine OS (e.g: ): Windows Kernel (e.g. ): 2019 Install tools: Network plugin and version (if this is a network-related bug): Others: Dockershim /sig windows /priority critical-urgent /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:


Question: if spec.Template.Spec.RestartPolicy != api.RestartPolicyAlways { allErrs = append(allErrs, field.NotSupported(fldPath.Child("template", "spec", "restartPolicy"), spec.Template.Spec.RestartPolicy, []string{string(api.RestartPolicyAlways)})) } <ins> if spec.Template.Spec.ActiveDeadlineSeconds != nil { allErrs = append(allErrs, field.Invalid(fldPath.Child("spec", "activeDeadlineSeconds"), spec.Template.Spec.ActiveDeadlineSeconds, "must not be specified")) } </ins> return allErrs }

setting activeDeadlineSeconds in deployment pod template, and seen new pods are continuously created after the older one dies. Is this excepted? if not we should forbidden this.
Yes, it is expected. Once a pod dies, the replica set controller will spawn a new one. activeDeadlineSeconds doesn't make any sense when used under a controller.
we may want to make it a validation error, but I don't feel strong.
Yes, we should not allow activeDeadlineSeconds for restartPolicy=Always pods generally.
What if I want a container to be restarted no matter what status it exits with, but to give up trying if it cannot even start up in seconds? So like this: Start a pod if it keeps constantly failing for seconds: stop it else (if it managed to start and not fail in those first seconds from creating): restart it everytime it fails or successfully finishes
then you want your own controller, RCs are designed for a simple use case: long-running apps. Have a look in , though keep in mind they are under a major redesign:
Sorta late to the party, but this behavior stops us from being able to use a Deployment to mitigate a production memory leak. Our application can handle a , but a leaves items in a pending state for our customers. Killing pods on a regular basis through kubernetes would allow us to induce known behavior of a graceful termination versus deploying new code to handle a timer before terminating. While I understand the intention of this feature, I feel that as a mitigation tool while my teams track down the memory leak, this would let my oncall rotation sleep easy.


Question: probe: v1.Probe{ InitialDelaySeconds: -100, }, <del> expectContinue: true, </del> <ins> expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> expectSet:      true, expectedResult: results.Success, },

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: MinimumGCAge:              minimumGCAge, Mounter:                   mount.New(), NodeStatusUpdateFrequency: nodeStatusUpdateFrequency, <ins> OOMAdjuster:               oom.NewFakeOOMAdjuster(), </ins> OSInterface:               osInterface, PodInfraContainerImage:    dockertools.PodInfraContainerImage, Port:              port,

The type has a field called to indicate that a container must run as non-root user. The description for this field includes the sentence , which would mean that setting the field to or to (unset) have the same effect: However, the PodSecurityPolicy provider implementation handles the (unset) case differently, and thus does differentiate between unset or values: The API type documentation should be updated to clarify this behaviour and reflect the actual PodSecurityPolicy implementation. /sig node
Thanks for reporting this issue. I think that would be best able to assist with this. /sig auth /remove-sig node
The pod documentation is correct. If the field is not set or is false, the enforcement is not performed by the kubelet. However, podsecuritypolicy with a runasuser strategy of mustrunasnonroot will force a pod that has not specified a runAsNonRoot setting or a runAsUser uid to runAsNonRoot: true The documentation on the MustRunAsNonRoot strategy of podsecuritypolicy should clarify the mutation behavior of the PSP /remove-kind bug /kind documentation
Can I take this issue?
^ this is something that can be changed.
Here's a proposed change to the documentation in : Is this sufficient to solve this issue? If so, should I submit a PR there?
I'd change that to to be a little more complete.
/assign See PR for docs at
GitHub didn't allow me to assign the following users: josiahbjorgaard. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign See PR for docs at Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
~This would have saved me some battling today! I couldn't work out why my pod wasn't selecting my run-as-root pod security policy and ignoring my restricted pod security policy - I didn't realise I had to explicitly set to get the correct PSP to match (I thought the restrictive PSP would reject the pod and then the run-as-root PSP would match, not that the restrictive PSP would mutate the PodSpec)~ It would have had I read the actual docs better sigh.


Question: /* Release: v1.19 Testname: Kubectl, server-side dry-run Pod <del> Description: The command 'kubectl run' must create a pod with the specified image name. After, the command 'kubectl replace --dry-run=server' should update the Pod with the new image name and server-side dry-run enabled. The image name must not change. </del> <ins> Description: The command 'kubectl run' must create a pod with the specified image name. After, the command 'kubectl patch pod -p {...} --dry-run=server' should update the Pod with the new image name and server-side dry-run enabled. The image name must not change. </ins> */ framework.ConformanceIt("should check if kubectl can dry-run update Pods", func() { ginkgo.By("running the image " + httpdImage)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Also flaking on: Which test(s) are failing: Since when has it been failing: ~Since Oct 21? Testgrid link: CAPG conformance: CAPA conformance: gce cos master default: Reason for failure: Example spyglass link: Anything else we need to know: Triage shows that this is happening across several jobs (not just the ones I listed above) and the spike started happening earlier this week. /sig cli /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fixed by
This is affecting ppc64le jobs as well.
/remove-kind failing-test /kind flake This is not a continuously failing test, it is instead flaking
/priority critical-urgent the top flake for per /area conformance this is a conformance test, it definitely shouldn't be flaking, and should be demoted if it can't be fixed
/reopen For CI signal to observe before marking as resolved
Reopened this issue. <detailsIn response to : /reopen For CI signal to observe before marking as resolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Last failure 7 days ago 2020-11-03 05:40:32 +0000 UTC
Please review to see if we can close this now.
Let's close this - also looks like it is no longer flaking as well


Question: } cleanup := func() { <del> if l.pv != nil || l.pvc != nil { By("Deleting pv and pvc") errs := framework.PVPVCCleanup(f.ClientSet, f.Namespace.Name, l.pv, l.pvc) if len(errs) > 0 { framework.Logf("Failed to delete PV and/or PVC: %v", utilerrors.NewAggregate(errs)) } l.pv = nil l.pvc = nil } if l.sc != nil { By("Deleting sc") deleteStorageClass(f.ClientSet, l.sc.Name) l.sc = nil } if l.volume != nil { l.volume.DeleteVolume() l.volume = nil } </del> <ins> l.cleanupResource() </ins> if l.testCleanup != nil { l.testCleanup()

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce Which test(s) are failing: I0221 20:48:18.462] [sig-storage] In-tree Volumes I0221 20:48:18.462] [ I0221 20:48:18.462] [Driver: gcepd] I0221 20:48:18.462] [ I0221 20:48:18.462] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode I0221 20:48:18.463] [ I0221 20:48:18.463] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources Since when has it been failing: unsure Testgrid link: Reason for failure: From the logs, it looks like it's deleting both PVC and PV objects at the same time, which is not correct. I0221 20:48:18.460] Feb 21 20:48:09.866: INFO: Deleting PersistentVolumeClaim "pvc-cjv2l" I0221 20:48:18.460] Feb 21 20:48:09.923: INFO: Deleting PersistentVolume "pvc--3619-11e9-b923-" should be changed to only delete PVC and wait for PV to be deleted by the driver. Can it just call which does it correctly? Anything else we need to know:
cc
It has been like that already before my recent code refactoring: Therefore I have to defer to - I myself don't really know what this test does or why it was written like that.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
/priority critical-urgent


Question: - createCoreV1Node - createCoreV1PersistentVolume - createStorageV1CSINode <del> - createStorageV1StorageClass </del> - createStorageV1VolumeAttachment - deleteCoreV1CollectionNamespacedPersistentVolumeClaim - deleteCoreV1CollectionPersistentVolume

[X] test approval issue : [X] test pr : [X] two weeks soak end date : 22 Sept 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three StorageClass endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following StorageClass endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/sig storage
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: //   colors: true // users: //   black-mage-via-auth-provider: <del> //     LocationOfOrigin: "" </del> //     auth-provider: //       config: //         foo: bar //         token: s3cr3t-t0k3n //       name: gcp //   red-mage-via-token: <del> //     LocationOfOrigin: "" </del> //     token: my-secret-token //   white-mage-via-cert: <del> //     LocationOfOrigin: "" </del> //     client-certificate: path/to/my/client-cert-filename //     client-key: path/to/my/client-key-filename }

yaml.Marshal a kubeconfig, as done in the example tests here LocationOfOrigin is not seen in output, as comment says "never serialized." Click "run example" doesn't suffer from this as it does some conversion <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig api-machinery
/assign /triage accepted
is what's known as internal type. It is not intended to be serialized by code. The serialization version is in and does not have this field. I could get behind removing all json tags from the internal version, but I wouldn't refine tags that shouldn't be used.


Question: return false } <del> func TestPlugin(t *testing.T) { </del> <ins> func getAzureTestCloud(t *testing.T) *azure.Cloud { config := `{ "aadClientId": "--aad-client-id--", "aadClientSecret": "--aad-client-secret--" }` configReader := strings.NewReader(config) cloud, err := azure.NewCloud(configReader) if err != nil { t.Error(err) } azureCloud, ok := cloud.(*azure.Cloud) if !ok { t.Error("NewCloud returned incorrect type") } return azureCloud } func getTestTempDir(t *testing.T) string { </ins> tmpDir, err := ioutil.TempDir(os.TempDir(), "azurefileTest") if err != nil { t.Fatalf("can't make a temp dir: %v", err) } <ins> return tmpDir } func TestPluginAzureCloudProvider(t *testing.T) { tmpDir := getTestTempDir(t) defer os.RemoveAll(tmpDir) testPlugin(t, tmpDir, volumetest.NewFakeVolumeHostWithCloudProvider(tmpDir, nil, nil, getAzureTestCloud(t))) } func TestPluginWithoutCloudProvider(t *testing.T) { tmpDir := getTestTempDir(t) defer os.RemoveAll(tmpDir) testPlugin(t, tmpDir, volumetest.NewFakeVolumeHost(tmpDir, nil, nil)) } func TestPluginWithOtherCloudProvider(t *testing.T) { tmpDir := getTestTempDir(t) </ins> defer os.RemoveAll(tmpDir) <ins> cloud := &fakecloud.FakeCloud{} testPlugin(t, tmpDir, volumetest.NewFakeVolumeHostWithCloudProvider(tmpDir, nil, nil, cloud)) } func testPlugin(t *testing.T, tmpDir string, volumeHost volume.VolumeHost) { </ins> plugMgr := volume.VolumePluginMgr{} <del> plugMgr.InitPlugins(ProbeVolumePlugins(), volumetest.NewFakeVolumeHost(tmpDir, nil, nil)) </del> <ins> plugMgr.InitPlugins(ProbeVolumePlugins(), volumeHost) </ins> plug, err := plugMgr.FindPluginByName("kubernetes.io/azure-file") if err != nil {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? NO Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes, azure file, chinacloudapi Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: What happened: We try to mount azure file to pod, and get the following error That is because azure china's storage DNS suffix is rather than What you expected to happen: Kubernetes should provide some method to let user specify storage dns suffix, so we can use Azure file volume How to reproduce it (as minimally and precisely as possible): Any Azure cluster in china can reproduce it. Anything else we need to know: kubernetes has hard coded in shows a relative issue
could you help confirm this?
One possible way is configuring DNS suffix via annotations of PV. If thinks this way okay, I'd like to create a PR to fix it.
azure file was implemented without using resource group or azure cloud provider. The endpoint can be found at One fix is to use azure cloud provider and choose the file service endpoint from azure environment as I did in blob service . Let me know if you can help.
I'm not familiar with azure cloud provider and volume codes. I'd like to try to dig into it to see if I can fix as the way you suggested.


Question: resp, err := rt.RoundTrip(req) if err != nil { <del> return nil, fmt.Errorf("error trying to reach service: %w", err) </del> <ins> return nil, errors.NewServiceUnavailable(fmt.Sprintf("error trying to reach service: %v", err)) </ins> } if redirect := resp.Header.Get("Location"); redirect != "" {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I upgraded from 1.15 and am using prometheus to scrape my pods via the proxy functionality. Previously no error message was being printed, now with 1.19.9 every try results in an error: Example url: https://x.x.x.x:6443/api/v1/namespaces/staging/pods/proxy-app--6sxfx:80/proxy/metrics What you expected to happen: No error message to be logged, since the apiserver anyways returns a corresponding http error code. The least thing I would expect it not to log as an error, but as a debug or info log, since it doesn't actually represent an error. It returns something like that: How to reproduce it (as minimally and precisely as possible): Call the proxy functionality on an endpoint that does not listen and will refuse the connection. Anything else we need to know?: There is a lot of other issues with a similar error message, but I believe they are not interconnected since this has explicitly todo with an error raised from the proxy functionality. Environment: Kubernetes version (use ): 1.19.9 Cloud provider or hardware configuration: on premise OS (e.g: ): CentOs 7 Kernel (e.g. ): 5.4.60
/sig api-machinery
/cc /triage accepted
/assign


Question: Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts metav1.PatchOptions, subresources ...string) (result *v1.ClusterTestType, err error) GetScale(ctx context.Context, clusterTestTypeName string, options metav1.GetOptions) (*autoscalingv1.Scale, error) UpdateScale(ctx context.Context, clusterTestTypeName string, scale *autoscalingv1.Scale, opts metav1.UpdateOptions) (*autoscalingv1.Scale, error) <ins> CreateScale(ctx context.Context, clusterTestTypeName string, scale *autoscalingv1.Scale, opts metav1.CreateOptions) (*autoscalingv1.Scale, error) </ins> ClusterTestTypeExpansion }

What happened: Test create method for non-namespaced subresources generate invalid code. I have the following code generation annotations: which generated test method with generated fake client: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted


Question: if err := w.handleCreateEvent(event); err != nil { klog.ErrorS(err, "Error when handling create", "event", event) } <del> default: </del> <ins> } else { </ins> klog.V(5).InfoS("Ignoring file", "path", path, "mode", mode) }

Looking at [1], we can see that the plugin Watcher is traversing the plugin directory, and check whether a new file is a unix socket or not. However, : Those file should not have been ignored. Instead of using, which also has a Windows implementation [2]. This issue may also cause the unit tests to flake [3]. [1] [2] [3] Instead of using, which also has a Windows implementation [2]. This will result in the plugin Watcher to work as intended. Run the ` unit test. No response Kubernetes v1.25, v1.24. N/A Windows Server 2019 or newer. <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /sig testing /milestone v1.25 /priority important-soon
/assign