Question: } return notMnt, nil } <del> // PathExists returns true if the specified path exists. // TODO: clean this up to use pkg/util/file/FileExists func PathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if os.IsNotExist(err) { return false, nil } else if IsCorruptedMnt(err) { return true, err } return false, err } </del>

What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?

Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,


Question: "k8s.io/component-base/configz" "k8s.io/component-base/logs" compbasemetrics "k8s.io/component-base/metrics" <ins> metricsfeatures "k8s.io/component-base/metrics/features" </ins> "k8s.io/component-base/metrics/legacyregistry" <ins> "k8s.io/component-base/metrics/prometheus/slis" </ins> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1" podresourcesapi "k8s.io/kubelet/pkg/apis/podresources/v1" podresourcesapiv1alpha1 "k8s.io/kubelet/pkg/apis/podresources/v1alpha1"

As per, , wire up metrics/slis for kubelet
/assign
/sig instrumentation
/sig node /triage accepted


Question: // NewEndpointSliceConfig creates a new EndpointSliceConfig. func NewEndpointSliceConfig(ctx context.Context, endpointSliceInformer discoveryv1informers.EndpointSliceInformer, resyncPeriod time.Duration) *EndpointSliceConfig { result := &EndpointSliceConfig{ <del> listerSynced: endpointSliceInformer.Informer().HasSynced, logger:       klog.FromContext(ctx), </del> <ins> logger: klog.FromContext(ctx), </ins> } <del> _, _ = endpointSliceInformer.Informer().AddEventHandlerWithResyncPeriod( </del> <ins> handlerRegistration, _ := endpointSliceInformer.Informer().AddEventHandlerWithResyncPeriod( </ins> cache.ResourceEventHandlerFuncs{ AddFunc:    result.handleAddEndpointSlice, UpdateFunc: result.handleUpdateEndpointSlice,

AKS had a customer report repeated issues in their clusters where: -proxy would redeploy (e.g. due to AKS deploying a new kube-proxy image with CVE fixes) c-ares DNS client would send repeated DNS queries to the kube-dns service VIP from the same src IP address, creating a UDP conntrack entry to the svc VIP and keeping it alive. For example, with kube-dns svc VIP 192.168.0.10, conntrack showed: queries to the svc VIP were blackholed until the customer manually ran a command like: Issue occurred only in clusters with many services and endpoints (~10k services and ~6k endpoints). However, the customer has seen this issue repeatedly for months, about 1-5 times per week across their clusters. kube-proxy code to delete stale UDP conntrack entries should have deleted the conntrack entry to kube-dns svc VIP automatically in the first sync after service and endpointslice cache is initialized. Unfortunately, the customer said they were unable to reproduce this issue in other environments outside of their production clusters. AKS engineers were also unable to repro this issue. Update (2024-08-03): I'm now able to reproduce the issue using a DNS client that reuses the same src IP / port and clears DNAT conntrack entries between queries, in a k8s 1.29.7 cluster with 2,000 services. The bug is triggered reliably in this setup with The test client code, scripts, and steps to reproduce are in this repository: The customer shared kube-proxy logs (at verbosity "3") from when the issue occurred. Unfortunately, I don't have permission to share the full logs publicly, but I think what I'm seeing in the logs gives a clue about what's happening. 05:00:02. nodes, endpointslices, and services all synced: -dns endpointslices processed after , but before iptables sync: iptables sync at 05:00:02. shows 4658 services, which is much less than the number of services in the cluster. (We can't see if kube-dns is one of these services, since kube-proxy at doesn't log which services were processed.)

at 05:00:02. and the next sync at 05:00:05., we see thousands of lines that look like services and endpointslices updates: sync at 05:00:05. shows 9605 services: I believe this could explain why kube-proxy does not clean up the stale UDP conntrack entry: kube-proxy detects that the endpointslice and service caches are synced using respectively. Comments on the say that this indicates that the informer cache is up-to-date, but warns "Note that this doesn't tell you if an individual handler is synced!! For that, please call HasSynced on the handle returned by AddEventHandler." This comment, along with the output of kube-proxy logs, makes me suspect that kube-proxy is initiating a sync before it has received and processed all the services and endpoints in the informer cache. If so, this could explain why the "delete conntrack" codepath is skipped. kube-proxy deletes stale conntrack entries for UDP services that are updated from and are in . If in the first sync is missing kube-dns, but includes kube-dns, then kube-proxy would . Subsequent syncs would see at least one endpoint for kube-dns, so would continue to skip deletion. If this theory is correct, then I wonder if kube-proxy could change the initialization check to ensure that all pre-sync events from the informer cache are delivered before the first iptables sync, maybe like this: Should kube-proxy wait until all pre-sync events are delivered before setting ? kube-proxy version is v1.28.5 Note that the kube-proxy image that AKS built for 1.28.5 includes the backported patch to fix this race condition in iptables partial sync: Azure Kubernetes Service Linux (I don't have the exact version but could get it if it's relevant) N/A N/A N/A
Possibly this issue would be addressed by
/sig network
you can workaround this in the meantime with an init container flushing all udp entries `
Just to clarify, an init container on the kube-proxy pod?

If I'm reading the code correctly, when kube-dns is missing from , then the iptables restore on the first sync won't install DNAT rules for kube-dns If so, I think even with an init container flushing conntrack, there's still a period between iptables restore and the next iptables sync where a DNS client could create a conntrack entry to the svc VIP.
I see, I don't have time to test it, but then you need to do two things up the kube-proxy iptables rules to avoid to have stale entries to services with endpoints that does not exist the existing UDP entries directed to Services a strawman approach to add to the existing kube-proxy manifest
joined us on the call today, and confirmed that we should move forward with the patch he proposed: /triage accepted /assign
I'm now able to reproduce the bug reliably using the test client and steps here: After applying the patch to wait for , the issue no longer occurs.


Question: Description: `By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs.` + "nn"}}, }, <ins> // SIGDescribe + KubeDescribe + It, Describe + KubeDescribe + It {"e2e/foo.go", ` var _ = framework.SIGDescribe("Feature", func() { KubeDescribe("Described by", func() { // Description: description1 framework.ConformanceIt("A ConformanceIt", func() {}) }) Describe("Also described via", func() { KubeDescribe("A nested", func() { // Description: description2 framework.ConformanceIt("ConformanceIt", func() {}) }) }) })`, []conformanceData{ {URL: "https://github.com/kubernetes/kubernetes/tree/master/e2e/foo.go#L6", TestName: "Feature Described by A ConformanceIt", Description: "description1nn"}, {URL: "https://github.com/kubernetes/kubernetes/tree/master/e2e/foo.go#L11", TestName: "Feature Also described via A nested ConformanceIt", Description: "description2nn"}, }}, </ins> // KubeDescribe + Context + It {"e2e/foo.go", ` var _ = framework.KubeDescribe("Feature", func() {

What happened: tried promoting a test with a tag in its full test description. The hierarchy of ginko-relevant AST's was . The tag was a in the top level . The full test description only came from , so the tag was missed. What you expected to happen: The conformance test list generator should have logged and ERROR and failed a test on the PR, calling out Disruptive as an invalid tag How to reproduce it (as minimally and precisely as possible): Check out locally and follow the directions in to generate the conformance list. The invalid test is in the list. Anything else we need to know?: Is there a function we can use to get the full test description from Ginkgo instead of rolling our own? If not, we should probably be maintaining a stack rather than just the last FYI who ran into this FYI who wrote the code to keep track of last
/sig architecture /area conformance
/milestone v1.16
/priority important-soon
/assign
GitHub didn't allow me to assign the following users: devaips. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil } <del> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, instances) </del> <ins> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, instances, service.Annotations) </ins> if err != nil { return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When multiple securityGroup are specified for AWS ELB type service, a random one will be chosen to grant inbound access to worker node. What you expected to happen: Always first LoadBalancer securityGroup be chosen to access worker nodes. How to reproduce it (as minimally and precisely as possible): use or on aws elb. Anything else we need to know?: Environment: Kubernetes version (use ): <=1.18 Cloud provider or hardware configuration: aws OS (e.g: ): linux Kernel (e.g. ): linux Install tools: kops/eks Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.

What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups.

We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster.

Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently.

I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).


Question: By("Trying to dial each unique pod") for i, pod := range pods.Items { <del> resp, err := http.Get(fmt.Sprintf("http://%s:8080", pod.Status.HostIP)) </del> <ins> body, err := c.Get(). Prefix("proxy"). Resource("pods"). Name(string(pod.Name)). Do(). Raw() </ins> if err != nil { Failf("Controller %s: Failed to GET from replica %d: %v", name, i+1, err) } <del> defer resp.Body.Close() if resp.StatusCode != http.StatusOK { Failf("Controller %s: Expected OK status code for replica %d but got %d", name, i+1, resp.StatusCode) } body, err := ioutil.ReadAll(resp.Body) if err != nil { Failf("Controller %s: Failed to read the body of the GET response from replica %d: %v", name, i+1, err) } </del> // The body should be the pod name. if string(body) != pod.Name { Failf("Controller %s: Replica %d expected response %s but got %s", name, i+1, pod.Name, string(body))

Some of the e2e tests specify HostPort in their pod configurations. This can cause issues running those tests if the port is already in use.
We've been removing these over the last week and a half and I think that may be the only one left remaining.
We need to decide if HostPort is a supported use case for Kubernetes -- and if it's supported, then it needs e2e test coverage to ensure it remains functional. Note that supported and not-preferred are orthogonal. My understanding is that it's supported-but-not-preferred and that there are reasonable use cases for using it.


Question: _ = client.CoreV1().ServiceAccounts(namespace).Delete("sample-apiserver", nil) _ = client.RbacV1beta1().RoleBindings("kube-system").Delete("wardler-auth-reader", nil) _ = client.RbacV1beta1().ClusterRoleBindings().Delete("wardler:"+namespace+":auth-delegator", nil) <ins> _ = client.RbacV1beta1().ClusterRoles().Delete("sample-apiserver-reader", nil) _ = client.RbacV1beta1().ClusterRoleBindings().Delete("wardler:"+namespace+":sample-apiserver-reader", nil) </ins> } <del> // A basic test if the sample-apiserver code from 1.7 and compiled against 1.7 </del> <ins> // A basic test if the sample-apiserver code from 1.10 and compiled against 1.10 </ins> // will work on the current Aggregator/API-Server. func TestSampleAPIServer(f *framework.Framework, image string) { By("Registering the sample API server.")

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: fi fi <del> KUBE_REPO_REL_ROOT="$(dirname ${BASH_SOURCE:-$0})/.." KUBE_REPO_ROOT="$(readlink -f ${KUBE_REPO_REL_ROOT})" </del> <ins> if [[ "$OSTYPE" == *darwin* ]]; then KUBE_REPO_ROOT="${PWD}" else KUBE_REPO_REL_ROOT="$(dirname ${BASH_SOURCE:-$0})/.." KUBE_REPO_ROOT="$(readlink -f ${KUBE_REPO_REL_ROOT})" fi </ins> KUBE_TARGET="${KUBE_REPO_ROOT}/output/go" mkdir -p "${KUBE_TARGET}"

What would you like to be : PV controller should not try to delete volumes that are attached. Right now, it tries to delete a volume as soon as PVC is deleted, which is too early. When user deletes a namespace, a pod and PVC is deleted at the same time, causing unnecessary delete attempts when the volume is still attached to a node. VolumeInUseProtection won't help us, as a Pod object is deleted before its volumes are detached. We've seen this in CI both on AWS and on GCE, I assume other clouds behave similarly. Why is this needed: It will reduce pressure on cloud API, as we try to delete the volume quite often (at least few initial retries are fast, until exp. backoff is high enough). On AWS, it would reduce our DescribeVolumes usage. It will reduce event noise / PVC being in Failed state. It will reduce logging noise. A simple implementation could pass Node informer into PV controller and PV controller could check before deleting a volume. The largest change would be actually in the unit test framework. /sig storage what do you think? Any volunteer for implementation? :-)
What if you put it in protection controller instead so we avoid having to make significant changes to PV controller?
Agree. Another idea I was thinking is now that we have volumeAttachment objects we could use those for synchronization between attach/detach controller and pv controller, but actually using is much simpler and works for all plugins, not just csi. As for putting it in the protection controller, I think it is achievable but it will be in a roundabout way. If we use protection controller, the logic in PV controller for when to delete a backing volume will be the same: if && ReclaimPolicy==Delete. I guess protection controller would somehow prevent from getting to Released until the volume is detached? If we use PV controller, the logic will change to be: if && ReclaimPolicy==Delete && PV is not present in any I don't think this is too scary/big of a change compared to changing protection controller. To the point "It will reduce logging noise:" we had a customer who saw these many "deletion failed because volume is still attached" and thought they were a "real" error.

Tried to give a technical explanation and they (and some other folks) were confused why pv controller and a/d controller don't communicate with each other...
Protection controller would need to keep the finalizer on until the volume is gone from volumesAttached. The way I see it, PV controller and attach/detach controller have independent concerns. Protection controller is actually coordinating operations between all these other controllers (ie kubelet and pv controller, AD and pv controller, etc)
OK. PV controller still needs to be aware of the finalizer somewhere before it calls . But I see your point about offloading the logic to protection controller.
The protection controller today already controls when PV controller goes into reclaim with the existing finalizer.
Is is it like: finalizer on the PVC prevents PVC from being deleted which controls whether PV goes into Released. Anyway, I can read the protection controller code on my own time... (Since I don't understand it I think that disqualifies me from implementing this? :))
Yes that's the gist of it :-) If you're interested, here's some leisure reading:
Is it possible that a volume may fail or take some time to detach from the host? If it has any chance, it might not a good idea to delay pvc deletion in this case. This will prevent users from using the pvc with the same name to claim a new volume or slow down this process. When using statefulset, pods have stable storage identities. If associated volume has failures or a new volume is required, new pvc can only be created after the old volume is detached.
I think this is not an issue for cloud volumes, which protect against deletion in its control plane, but for other plugins that don't have this kind of protection, would deleting the volume first cause detach to permanently fail?
Would need metrics to prove this, but my suspicion is that implementing this would actually improve pvc deletion time. Currently, deletion fails repeatedly while the volume is still attached and it exponentially backs off. Then the next step after the volume is detached succeeds. With this implemented, deletion should succeed on the first try almost immediately after the volume is detached.
Right now pvc can be deleted immediately if the pod is gone (pvc protection controller). It’s not possible to improve pvc deletion time. I might be wrong.

Volume is going to be released and reclaimed (Delete reclaim policy) after pvc is deleted. If we are going to solve this issue in protection controller, pvc deletion should be delayed until volume is detached. My concern is this might slow down or fail pvc deletion if the volume has some issues to be detached from the host. And I think the problem of old volume should not prevent users from requesting a new volume.
For cloud volumes, deletion will be triggered, but will fail repeatedly until volume is detached because the cloud control planes prevent disks from being deleted while attached.
I see. Sorry I don’t know much about cloud volumes.
I agree this problem should be solved but don’t think it is a good idea to delay pvc deletion to delay volume deletion. I remember there are some issues to detach volumes from unresponsive nodes (because we don’t know the node is shut down or only inaccessible from apiserver). I prefer implementing it in pv controller.
IMO, it's only clouds who implement 3rd party attach - because there must be some control over the hosts (real HW, not necessarily VMs) to add / remove devices there. And those typically have a protection. The reason we have PVC protection is that users tend to delete PVCs while they still have running pods. They then get surprised that their PV is deleted and their pod is not happy that underlying storage is gone (e.g. iSCSI starts returning I/O errors). IMO it's a good idea to extend the protection from "being used by a pod (=being mounted)" to "being attached". Because the volume should not be deleted anyway. (On related note, I've seen AWS API taking detach for more than 2 minutes when detaching thousands of volumes at once. Damn API throttling).
PV can wait for its underlying storage to be detached from the host but IMO pvc does not need to. Otherwise a pvc cannot claim (recreate with the same name) a new PV until previous PV is detached from the host.
We encountered the same issue in our production environment.

In our env, we may immediately delete PVC once the pod is deletedand the following two workflow will happen: delete event -attach detach controller -(VolumeAttachment)-csi attacher controller -csi plugin -cloud volume detach open api, and it will take about 3~5s normally delete event -pv controller set PV into status -csi provisioner controller -csi plugin -cloud volume delete open api, and it will take less than 3s normally So the operation of deleting volume is very likely to be executed before volume is detached successfully when pod's and pvc's deletion events are triggered at the same time. How did we avoid it? We do check if the volume is fully detached at each deletion operation in csi plugin, and do real deletion after detachment, but which led to many retries for volume deletion operation.
IMO, firstly, pvc&pv protection controllers is used for preventing that pvc&pv resource object themselves are deleted from etcd when they are used or referenced, but don't care the under operation sequence of volume, secondly, it's not a good idea about judging if volume has been detached via , because AD controller updates before triggering real volume detachment action in .
And my idea to fix or improve this issue is that AD controller add new finalizer to PVC after bound PV is attached to one node then removes it after bound PV is detached from all node. This new finalizer can act as the communication medium between AD controller and PV controller.
Will this cause any surprises for users who previously managed to delete PVs from unresponsive nodes and now they can't?
instead of trying to solve this for all volume types (considering that most of the attachable ones are getting migrated to csi), what if we look into solving this just for CSI, and use VolumeAttachment object?
Actually external-attacher already seems to until all VolumeAttachment objects are gone. I guess this doesn't work because external-provisioner ignores it?
External-attacher finalizers are not evaluated in the provisioner. Maybe they could. It introduces dependency between the attacher and provisioner, but I don't see a big issue there. The provisioner could even check if there is a VolumeAttachment object for a PVC before deleting it.
/assign
GitHub didn't allow me to assign the following users: RaunakShah. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned.

Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is interested in working on this.
Thanks /assign are we okay with either of these?
I would prefer to check for existence, it looks cleaner from API point of view.
Hm that means external-provisioner needs to watch and lookup VolumeAttachment objects?
Checking PV finalizers will work, but basing API on finalizers looks quite odd.
I see that external-provisioner only has a PersistentVolumeClaimLister, but not a PersistentVolumeLister. So we'll need to add a Lister to watch PersistentVolume objects too if we check finalizer on PV's?
For a multi-attach volume, you can have multiple VolumeAttachment objects for the same PV on different nodes. Would external-provisioner be able to properly serialize without races in this case?
I prototyped adding a PersistentVolumeLister and checking for external-attacher Finalizer before calling here - Albeit with fairly primitive testing, it seems that this creates a loop of dependancies between external-attacher and external-provisioner: -provisioner will not call until the Finalizer prefixed with "external-attacher/" is removed. -attacher does not remove its Finalizer until is set on the PV.
I think so. At the time is called, PVC was deleted, so no new VolumeAttachment can be created (unless someone creates it manually). No, call from the provisioner lib gets PV: It gets an older instance of the PV (, from informer), while the provisioner has already a newer instance (, from GET), but that should not hurt - in the worst case, gets a PV with finalizer and returns an error and new will follow after exp. backoff. And the provisioner lib can be fixed easily. Good catch! I totally forgot about this. Waiting for prevents the attacher from races between removing the finalizer and at the same time a new VolumeAttachment being created.
Would appreciate your reviews at Based on the discussions above, i've gone with the volumeattachment approach.

If you feel that there's a potential race condition that needs addressing too, please let me know!
Any new updates about this issue?
PR is being reviewed here:
We have a customer running into detach/delete race with the in-tree plugin. The fix is only available for CSI customers. Can we fix this issue for the in-tree volume plugins as well? Customer is not yet ready to move the production workload to CSI. cc:
cc


Question: ginkgo.It("should verify ResourceQuota's multiple priority class scope (quota set to pod count: 2) against 2 pods with same priority classes.", func() { _, err := f.ClientSet.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: "pclass5"}, Value: int32(1000)}, metav1.CreateOptions{}) <del> framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) </del> <ins> if err != nil && !apierrors.IsAlreadyExists(err) { framework.Failf("unexpected error while creating priority class: %v", err) } </ins> _, err = f.ClientSet.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: "pclass6"}, Value: int32(1000)}, metav1.CreateOptions{}) <del> framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) </del> <ins> if err != nil && !apierrors.IsAlreadyExists(err) { framework.Failf("unexpected error while creating priority class: %v", err) } </ins> hard := v1.ResourceList{} hard[v1.ResourcePods] = resource.MustParse("2")

The tests are full of assertions which compare a boolean against true or false, often without any additional explanation. When those assertions fail, the error message is useless for understanding what went wrong, often just saying: Including an explanation is better, but then we still have the distracting noise about the comparison: All of those assertions should be replace with . Such locations can be found by grepping: UPDATE (2024-08-26): in cases where such an if check is not possible (typically in combination with or ), should be used instead of . They emit only the explanation given to them, without including the noise above. The Ginkgo linter (enabled in golangci-lint) points that out. Some progress has been made in the meantime: v1.23.0: 250 v1.24.0-beta.0: 177 However, we were down to 175 at some point, so the number is going back up. Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: It makes sense to split this work into individual PRs, one per OWNERS file under test/e2e. Here's a list of directories: [ ] test/e2e/apimachinery [x] test/e2e/apps: [x] test/e2e/auth [x] test/e2e/autoscaling: [x] test/e2e/cloud/gcp [x] test/e2e/cloud [x] test/e2e/common/network [x] test/e2e/common/node: [x] test/e2e/common/storage: [x] test/e2e/framework: [x] test/e2e/framework/providers/vsphere [x] test/e2e/framework/volume [x] test/e2e/instrumentation/logging [x] test/e2e/instrumentation: [x] test/e2e/kubectl [x] test/e2e/lifecycle [x] test/e2e/node: [x] test/e2e/scheduling: [x] test/e2e/storage: (prior PR: ) [x] test/e2e/testing-manifests/storage-csi [x] test/e2e/upgrades/apps: [x] test/e2e/windows: , When creating a PR, use as title. Use one commit per PR, squash when addressing feedback.

Unless noted otherwise, will work on PRs. Help may be welcome. /sig testing /help Better output for developers when tests fail.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: /sig testing /help </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/triage accepted
I would like to take this up as my first issue
/assign
I would like to take up this issue as my first issue
Hello I am already working on this issue , can you please take some other issue ?
Okay sure ,how do I un assign myself from this issue?
I don't think you are assigned to this issue.
Oh okay
Hey as discussed in pr I am assuming that you are only working for tests under area.

also finished the work on , see
and only one PR is to be made for the correction under test/e2e/storage/, so how can we distribute the task and also make only single PR. Or can we make multiple PR's
I have a TODO list to the issue description. I'll try to keep that up-to-date with what has been done and who is working on it.
you can continue with in
what is to be changed in this case !
can you inform on which directory you are working.
Currently working on apimachinery
Nothing, just leave it as-is.
But that isn't under - before starting somewhere else, please coordinate with
test/e2e/instrumentation/logging don't have any cases like "Expect.Equal([^,], *(true|false)", so I think it should be removed from tasks to do.
I am thinking that the output should be if !ok { framework.Failf(uexec.ExitError) } in this case, am I thinking reasonable here.
isn't valid Go. The first parameter must be a format string. The code checks whether err is a : Therefore the change should be:
Why don't we just add extra explanation into calls like for easy investigation? I might miss some point.
You have to consider the complete output when that case fails. Gomega will write print the actual value and the expected value, then in addition you get the same information in the custom message. Printing just the custom failure message is shorter and more readable, in particular for a comparison of booleans. For a comparison of integers I see two solutions: ExpectEqual with a fixed message that just explains what is being compared. An if check with Failf where the message includes the actual value. The later seems more useful when the expected value is zero because that value doesn't need to be printed when its obvious.
I am getting difficulty in figuring out what should be output here, what should the appropriate output here. Also what steps you took in knowing so, it would help to improve my understanding.
The "ok" value is the result of checking for a specific key in a map. What a developer therefore wants to know when that check fails is: what was the key what's in the map Both can be included in:
can I work for the section :- test/e2e/scheduling ?

And there was only one changes needed for test/e2e/node which I did but my PR was closed as it only contains trivial edits. That was my first PR, so can you please guide me further?
/assign I'll try to coordinate the work, but won't create PRs myself. I ticked of line items which had no remaining work. feel free to start working on .
I replied there and also suggested an improvement. For those following this issue: try to include more information than before in the error message when the assertion fails. Then it is more obvious to the reviewer that the change is useful without having to read this issue. Examples:
Another tip: don't just reference this issue as explanation for the PR. Reviewers are busy, they won't necessarily invest the time and follow the link to understand what the PR is about. Instead, take the first sentence from this issue and adapt it as needed to the specific changes in the code.
can I start working on test/e2e/common/storage ?
Okay.
for test/e2e/network/netpol , there are already message associated with all of the cases, for example :- So should I just change all of them to ? And same for all the cases in test/e2e/network except these two cases : framework.ExpectEqual(netutils.I sIPv4String(internalIPs[0]) != netutils.IsIPv4String(internalIPs[1]), true) framework.ExpectEqual(netutils.IsIPv4String(p.Status.PodIPs[0].IP) != netutils.IsIPv4String(p.Status.PodIPs[1].IP), true )
Yes.
Okay then I will be working on and
It might make sense to combine them in a single PR, depending on how many changes there are - just use common sense.
IMO : this will never fail because whether err will be nil or it will exist. So, we don't need to change this code. What I had done in this case was :
Note that that is redundant (= not wrong, just not needed). I later clarified that in
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Hi! I've created a PR to address the tests in e2e/apps It is my first PR so hopefully, everything is ok. Thanks a lot!
Hi can I take test/e2e/common/node?
Yes, please do.

Have we considered / helpers that can skip the comparison output? - is noisy and makes it harder to skim tests, especially given that it's a pretty commonly required pattern.
No, we haven't. Do you find it noisy because it's two lines instead of one? I don't have a problem with that. It seems to be a common approach in unit tests and I find it easy to read, partly because it matches how production code would be written.
It becomes three, plus extra indention - but more importantly stands out in a series of expectations. This draws extra scrutiny to those lines. many unit tests using testify would use to express this because of that.
Isn't that a good thing? is bad without any or no useful extra explanation. The if check makes it obvious that a good failure message is needed. Does print less useless boilerplate code than Gomega? We could use that with Ginkgo (), but then we still need to ensure that the optional explanation is useful.
xref
Hi can I take test/e2e/auth?
for test/e2e/auth, for example : Should I just change it to ?
feel free to work on test/e2e/auth. This one particular example is a bit different: here we are comparing string values, and for those it makes sense to print the actual value because the failure message then adds information. I would leave that one as it is.
OK. test/e2e/auth doesn't need replacing. I : <img width="731" alt="Screen Shot 2022-05-30 at 9 12 59 PM" src="https://user-"There are a lot of positions that need to be modified in . I'll take test/e2e/apps.
That looks like a good candidate. Please go for it. When you need to make up new failure messages, try to think about what information may be useful to have and add that. Then the resulting PR is clearly better than the status quo.
Hi can you please assign me a directory. I want to give it a try. Thanks,
Hi I already posted a PR for e2e/apps some days ago in this issue: If it is fine I'd love to proceed with that. Just to avoid us repeating work.
sorry for not noticing the overlap sooner. Let's go with the older PR.

you can take over test/e2e/storage from There is an incomplete PR for it in You can check out that branch, fix it and submit a new PR, or start from scratch.
it looks like is not active anymore. I've remove him from the todo list and you instead for test/e2e/common/storage.
OK. Thank you. Is there an incomplete PR for .
I in and it returns 0. What should I do?
needs modifying. <img width="1440" alt="Screen Shot 2022-05-31 at 2 40 29 PM" src="https://user-"Can I work on it?
Indeed, did test/e2e/common/storage in You can work on test/e2e/windows.
OK. I'm working on it currently.
Sure, I will look into it. Thanks,
: do you want to continue with test/e2e/apimachinery?
Yes. There's an older PR:
Hi is there anything I could take over?
Hey I am not sure how updated the list is, but I am available to help. Is there any part I could start or continue?
sorry for the late reply. I had (and still have) some big changes for the E2E tests going and had put this on the backseat for a while. If you are interested, how about doing test/e2e/apimachinery and test/e2e/autoscaling? Some work was done on the latter, but there are still several calls (again?).
Sure, I can take that!
Hey I'd also like to contribute on this issue. Is there anything I could take over?
feel free to take my part:
Perfect, then I will take part
Hi :smile: , I would like to contribute to this issue, there is anything that I could do?
I was reluctant to start new work on this because I had one big PR pending that touched a lot of test code (PR ), but that has been merged. you are taking test/e2e/autoscaling, right? test/e2e/apimachinery? you can do test/e2e/apps. That leave test/e2e_node in case someone wants to continue.
yes e2e/autoscaling. I had some trouble to get the tests going initially, but seems fine now!
Hi, Can I do something to contribute to this issue?

you could work on test/e2e_node.
Ok, I will start work on test/e2e_node thanks!
Hi , I would like to contribute. Is there anything I could take over?
thanks for your interest, but right now everything seems to be covered.
Hi Thanks. I am searching for "good-first-issue", feel free to assign one to me. I am exploring api-machinery, but happy to contribute in other sig as well.
maybe you can take a look at if you have some time :)
Hi , There is anything that I could do to contribute with this issue?
Hi , I was looking and found the test/e2e/apimachinery without doing it and only then did I see that it was already assigned if this is a problem just refuse my pr and if you have something else to do let me know.
The remaining work will probably finished in , so there isn't anything else to do. If someone wants to investigate writing a linter for this, then this would be a cool science project - not easy, though. I would start with looking at for this. It has the advantage that it is supported indirectly in golangci-lint ().
Hello I would like to try to do a liter for it. It will be a cool project whit a lot of opportunities to learn. There will be a separate issue for the linter or we continue the discussion here?
Let's continue here.
I ended up creating a separate issue for linter ideas: let's continue there.
Ok!
New tests have been again with this anti-pattern since the last time that I checked. It's clear that without a linter, we'll not get rid of it. So let's focus on that first.
is this issue still open
Yes, this is still open. Current status as of v1.28.0-rc.0: Two calls are in , the rest are in . We have such a linter now, with ginkgolinter in golangci-lint and (soon) the ability to prevent usage of , so it makes sense to continue with removing those calls. do you want to prepare one PR for all remaining calls? I think one PR is enough, no need to split it up.
also has it, which was intentional. I'm undecided whether that should be replaced.

Several of those calls are being touched in , let's wait for that PR to be merged before doing any further work.
no longer finds anything. We might be done... :tada: But not quite. What we have left now is checking a boolean via or , which has the same problem. In some cases, there is an extra message (which is still noisy, see ), in other cases there isn't even that (which makes the failure message fairly useless). We can and should replace some of those calls with the pattern.
is almost resolved. My work-in-progress branch forbids the usage of in favor of the new or plain Go. In some cases, moving the Go code that returns true/false into a matcher might be the best solution because that allows dumping the full actual value that was being checked. Here are the remaining issues: <details</detailsDoes anybody want to work on those?
I would love to!
: please go ahead. Assign PRs to me when you create them. Let's start with a few of those calls initially, do a round of reviews, then continue. /assign
You will have to start with my branch, current master doesn't have gomega with thew new API yet.
I'm starting new to kubernetes. Is there anything easy that I could do ?
Hey Is there anything left I can help you with in this issue ?
Thanks for your interest. There's not much to be done at the moment, though.
Should this be closed?
Oh wow! Looks like we are indeed done. Thanks everyone. /close
Closing this issue. <detailsIn response to : Oh wow! Looks like we are indeed done. Thanks everyone. </details


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package workqueue import "time" type TimedWorkQueue struct { *Type } type TimedWorkQueueItem struct { StartTime time.Time Object    interface{} } func NewTimedWorkQueue() *TimedWorkQueue { return &TimedWorkQueue{New()} } // Add adds the obj along with the current timestamp to the queue. func (q TimedWorkQueue) Add(timedItem *TimedWorkQueueItem) { q.Type.Add(timedItem) } // Get gets the obj along with its timestamp from the queue. func (q TimedWorkQueue) Get() (timedItem *TimedWorkQueueItem, shutdown bool) { origin, shutdown := q.Type.Get() if origin == nil { return nil, shutdown } timedItem, _ = origin.(*TimedWorkQueueItem) return timedItem, shutdown } func (q TimedWorkQueue) Done(timedItem *TimedWorkQueueItem) error { q.Type.Done(timedItem) return nil } </del>

The TimedWorkQueue object appears to lack a critical feature of the original WorkOueue, namely that similar items are aggregated together. By adding the time member to to the key type, it ensures that (time, item) pairs are unique-- you can add the same item multiple times as long as the time differs. This removes the guarantee that two goroutines wouldn't process the same item at the same time.
Iirc, the timed queue feeds a regular queue, so the guarantee still holds
Sorry, I was thinking of the delaying queue, which does key correctly
I'm going to remove timed workqueue. It's used to gather metrics for queues used in the garbage collector. This feature is supported by a regular workqueue now.


Question: ### Alternatives <del> A contributed [example](coreos/coreos_multinode_cluster.md) allows you to setup a Kubernetes cluster based on [CoreOS](http://www.coreos.com), using EC2 with user data (cloud-config). </del> <ins> CoreOS maintains [a CLI tool](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html), `kube-aws` that will create and manage a Kubernetes cluster based on [CoreOS](http://www.coreos.com), using AWS tools: EC2, CloudFormation and Autoscaling. </ins> ## Getting started with your cluster

Our docs say this: I think though that example is out of date (?). We should make sure to point it to the right place.
cc
where should this point?
The official CoreOS guides are documented/linked here: The decision made a few months ago was to leave the previous community content, but should we revisit that?
Side note, a link to our AWS tool/guide probably needs to be there as well
What file contains that sentence?


Question: }, }, }, <del> expected: withVendorExtensions(new(spec.Schema), "x-kubernetes-preserve-unknown-fields", true). Typed("object", ""), </del> <ins> expected: withVendorExtensions(new(spec.Schema), "x-kubernetes-preserve-unknown-fields", true), </ins> }, }

What happened: I have CRD with a spec field of type and annotation . I want to store json data in this field which is actually a Grafana Dashboard json. When I try to create a CR of this type, I am getting validation error from json bytes. . In the json data . What you expected to happen: I expected no error since I set How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.17.0 Cloud provider or hardware configuration: kind OS (e.g: ): Ubuntu 18.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig api-machinery /area custom-resources
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: I have CRD with a spec field of type and annotation . I want to store json data in this field which is actually a Grafana Dashboard json. Kubernetes version (use ): v1.17.0 Cloud provider or hardware configuration: kind OS (e.g: ): Ubuntu 18.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig api-machinery /area custom-resouces Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:
, you are able to create the CRD, which is also working for me. Are you able to create the CR for this CRD? Does the following command work for you?
Running the second command:
I think you have not read the issue clearly. The CRD creation is working fine. I am getting validation error when trying to create CR for this CRD.
I edited the previous comment with result for second command.
Thanks ! This is the issue I see. I expected no error since I set .
If you pass --validate=false, does it create successfully?
Yes, if I pass --validate=false, it creates successfully.
Then the issue is in client-side kubectl validation, and the server is behaving as expected
/cc for openapi validation
Yeah, we should probably make anything that has the accept anything.

I thought we already did so for types that are . Maybe this never landed actually, I could look it up.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
That's probably not super hard to solve if someone wants to try.
A good way to start would be to solve all the TODOs in Or possibly, to address this bug specifically: This is the function that validates the "kind" (i.e. structures): When I initially wrote this code, we assumed that these type would never have additionalProperties. We probably need to change this condition since it's not really valid: and this one: We create the kind here: That function would have to be changed to save the "additionalProperties" type (if given) or the ability to have additionalProperties.
Actually, the problem also is that it doesn't know how to handle null. This one is a little tricky.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I think the publishing of the "object" type must be supressed for fields. We have similar cases already.

Touching kubectl is not very effective, or at least it takes multiple releases to makes its way through the ecosystem.
/assign


Question: if pod.DeletionTimestamp == nil || kubetypes.IsMirrorPod(pod) { return false } <del> // Delay deletion of pods until the phase is terminal. </del> <ins> // Delay deletion of pods until the phase is terminal, based on pod.Status // which comes from pod manager. </ins> if !podutil.IsPodPhaseTerminal(pod.Status.Phase) { <del> klog.V(3).InfoS("Delaying pod deletion as the phase is non-terminal", "phase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </del> <ins> // For debugging purposes we also log the kubelet's local phase, when the deletion is delayed. klog.V(3).InfoS("Delaying pod deletion as the phase is non-terminal", "phase", pod.Status.Phase, "localPhase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </ins> return false } // If this is an update completing pod termination then we know the pod termination is finished. if podIsFinished { <del> klog.V(3).InfoS("The pod termination is finished as SyncTerminatedPod completes its execution", "phase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </del> <ins> klog.V(3).InfoS("The pod termination is finished as SyncTerminatedPod completes its execution", "phase", pod.Status.Phase, "localPhase", status.Phase, "pod", klog.KObj(pod), "podUID", pod.UID) </ins> return true } return false

We observed the following series of events: 1) Pod is created with some attributes that later result in the pod being rejected at kubelet admission time 2) The pod is scheduled to a node 3) The pod is gracefully deleted by a controller, i.e. a deletion timestamp is set on the pod 4) The pod is rejected at kubelet admission time 5) The pod never gets deleted and is stuck: despite having the deletion timestamp set. It seems that kubelet does not issue the final force delete for the pod. This is problematic in many scenarios, but one specific case it was hit was when the pod is backed by a daemonset controller. This issue manifested in way where since the pod never got deleted in step 5 above, the daemonset controller did not create a new pod to run the on the node and thus node was was not running a replica of the daemonset controller pod. This seems to a be regression in k8s 1.27, I was not able to repro this behavior in 1.26. It is expected at during the series of events above, after the pod is failed at kubelet admission time, since the pod has a deletion timestamp set, the kubelet should forcefully delete the pod. The pod should not hang in deletion forever. Create a 1.27 kind cluster: Stop kubelet Create a pod bound to the node which will trigger a admission error. The below node selector label does not exist, so we expect that the pod should be rejected in admission. While kubelet is down issue a graceful deletion to set a deletion timestamp on the pod Verify deletion timestamp is set (pod is pending) Start kubelet Observe pod status, the pod is now in failed phase (as expected). However, the hangs, the pod is never force deleted by kubelet. No response 27 <detailsn/a </details<details</details<details</details<details</details<details</details
/sig node
cc
The kubelet logs have the following in the above repro: and there is no log about the pod being deleted by the status manager. Full kubelet logs - The log seems unexpected :) - the pod is not being deleted because the phase is "non-terminal", but the phase is failed which is a contradiction.

It points that in the code something is wrong because
/assign I've checked that this is regression caused by this (the scenario works when this PR is reverted on 1.27.2). One relevant difference is that the function now returns false, because the . However, even if we make it return , still the deletion will not happen because . Wondering about the preferred solution. I guess ideally if we could trigger the . OTOH it wasn't triggered before and a pod rejected due to admission does not really need termination of containers, so maybe a shortcut will work. FYI
Opened PR:
Sigh... then it sounds like the daemonset controller is not handling correctly pods with a terminal phase. It looks like its waiting for them to disappear. Can you open another issue for that?
The pod that revealed the kubelet regression was indeed created by DaemonSet. However, this issue is reproducible without DS. Moreover, I cannot reproduce it with Kubelet fixed.
But this is the same problem as The DS controller should react to a Pod in a terminal phase and create a replacement Pod. I agree, if we fix kubelet, it's a non-issue, but the DS controller is not behaving correctly.
In this scenario the pod already has a , so DS ignores it waiting for it to complete (which would normally happen if Kubelet works ok): Not sure what is the expected behavior of DS here. I guess DS could look at what is the expected , and if exceeded substantially (like a minute or two) then force-delete the pod itself.
DS should consider the Pod terminal and, as such, create a replacement. Note that I'm referring to a state of the Pod where it has a terminal phase, regardless of whether it has a deletionTimestamp.
I see, going to open a dedicated DS Issue to move the discussion on priority and the shape of the fix there. Let's focus here on the Kubelet.
/cc
/triage accepted /priority important-soon
/cc
A pod that has been rejected due to admission should be continually readded to the pod worker by HandlePodCleanups until it succeeds at being terminated. That's because the pod is not in the desired set. Is HandlePodCleanups not adding it it in "Kill" state to the pod worker?
I took a closer look at this and is not handling this case.

This is because the pod has been rejected at kubelet admission time, so a pod worker has not started for it. Additionally, the pod is not visible in the runtime, so it's not an orphan pod either. The main reason is because the pod has never started, so there is no pod worker driving it. The pod is returned from the desired state, pod manager (as part of . However, the pod is filtered out as part of because there was never a pod worker driving it, so there no for it. And since the pod is not part of and it's not an orphan pod (since it doesn't exist in runtime because we haven't started it), we end up not do anything with it in .
Should it be? I think right now our answer is no - but if we moved admission closer to the pod worker, we'd say yes? Pros and cons? We start pod workers for terminating a pod - is rejecting a pod a type of rejection? For the short term, why wouldn't HandlePodCleanups manage this case? It's the "resync" action for the kubelet as controller, and so it's nominally responsible for forcing all other components (pod worker, status manager, runtime) to return to state. Should it dispatch a TerminatePod call for these rejected pods that are not yet deleted, not known to the pod worker, and not known to the status manager?
Long term this might be the best option - having every path go through pod worker and the regular pod termination (i.e. instead of special casing special situations like admission rejection, may be simpler to reason about. Adding this to is definitely one option and I think we could expand it to cover this case. The only issue I see handling it there, is there could potentially be a large number of pods that are in terminal phase, not have a deletion timestamp set, and unknown pod worker and pod runtime. For each of these pods, we would have to call in every iteration of until someone eventually set the deletion timestamp and the pod would be deleted. Looking at the code, it should short-circuit , but we would need to confirm. Me and were also considering two other options: during kubelet rejection.

I think it should be fine overall, but has a risk that after a kubelet restart, if a previous pod was still in runtime and it's running, could lead to the pod being deleted by status manager before the orphaned pod is terminated. by pod worker during kubelet rejection. It should go through pod worker and check that all containers are terminated, so it should avoid issue of the first approach. It sounds like option 2 or adding this HandlePodCleanups is the optimal paths forward. Curious to get your thoughts on the options here.
/retitle Pod graceful deletion hangs if the kubelet already rejected that pod at admission time There are 2 admission phases for Pods (API server admission, kubelet admission). This issue is about the second phase only.
Hi, are there any workarounds in 1.27?
The latest version of GKE contains a mitigation for this issue when it's related to Spot VM preemptions. It's still rolling out in some regions/zones. Please ensure you are in the latest patch version available in GKE.
The fix was cherry-picked for the 1.27 branch in , so it should be included in 1.27.4.
; we're on the latest patch () and we see this.
Can you clarify what you observe? We fixed a few different things that were related. Some of the fixes should be available in 1.27.3 and the last one in 1.27.4.
Hey no problem, we upgraded our cluster from to today, a new node came up and 37 pods were scheduled to it. One of those failed with: We've never seen this before in the 3+ years we've used GKE and upgraded in this manner, so appears to be related to the 1.27 release. I assume you work for Google? If it would help you I can raise a support case and give you the case number so you can inspect our logs.
That sounds unrelated to this issue. Pod failures (even if uncommon) are expected in a distributed system such as kubernetes. What's important is that the system should recover. This usually means that a replacement pod is created. In your case, was a replacement pod not created? (for Pods backed by Daemonset, StatefulSet, Job, Deployment).

Nevertheless, just the error message you describe sounds more like , not this issue.
I am getting at 1.27.4-gke.900 running at GCP.


Question: // GetPowerStatusByNodeName returns the power state of the specified node. func (ss *scaleSet) GetPowerStatusByNodeName(name string) (powerState string, err error) { <ins> managedByAS, err := ss.isNodeManagedByAvailabilitySet(name, azcache.CacheReadTypeUnsafe) if err != nil { klog.Errorf("Failed to check isNodeManagedByAvailabilitySet: %v", err) return "", err } if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetPowerStatusByNodeName(name) } </ins> _, _, vm, err := ss.getVmssVM(name, azcache.CacheReadTypeDefault) if err != nil { return powerState, err

What happened: When the standalone (or VMAS) instance backing a registered node is removed (from Azure, ie. the instance is deleted), that node will remain forever due to the Azure cloud provider only checking whether the node was removed from a containing VMSS. Moreover this will triggers frequent VMSS full re-scans (when the instance name conforms to the expected VMSS instances name format, as we cache miss on every call), hammering Azure's VMSS List API, and causing throttling. A VMAS/standalone node leaving a cluster would cause controller-manager to log that error (here on k8s v1.18.2): What you expected to happen: The cloud provider should answer CM node lifecycle controller that the VM instance went missing and the node can be garbage collected. How to reproduce it (as minimally and precisely as possible): 1- Create a non-VMSS Azure node 2- Configure it to register to an Azure k8s cluster 3- Delete that node's instance from Azure console Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.2 Cloud provider or hardware configuration: Azure, controller-manager running with and OS (e.g: ): Ubuntu 20.04 LTS Kernel (e.g. ): Linux 5.4.0-1010-azure Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure


Question: <-executed t.Logf("runDelegationTest received executed signal at " + time.Now().String()) <ins> wg.Wait() t.Logf("runDelegationTest nested decorators finished at " + time.Now().String()) </ins> } func TestProc_doWithNestedX(t *testing.T) {

Can we get this fixed ASAP? Thanks!
yep, thanks for reporting , Brendan Burns wrote:
how reproduceable is this? I have some changes that I think will resolve it but I couldn't reproduce locally even when all my cpu's were pretty tied up and using a high GOMAXPROCS.
flakey label process?
the labelling is due to me failing @ github's label interface. Bulk editing labels removes them if any of the selected things have the label sigh Thanks for the quick fix.


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", <ins> "//vendor/github.com/stretchr/testify/assert:go_default_library", </ins> "//vendor/k8s.io/utils/pointer:go_default_library", ], )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The cluster started giving a lot of NODE DENY error in the log corresponding to restarting kube-apiserver make another one crash in a 3 apiserver setup. (say systemctl restart kube-apiserver on controller-1 will crash controller 2 or 3) It's as if node were show as but were not registered anymore?! What you expected to happen: The cluster to work as it did the weeks before How to reproduce it (as minimally and precisely as possible): I haven't fixed this yet and pod are stock in pending most of the time even after restarting controllers, workers, etc. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 10 workers with 64GB on 3 controllers VM. OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.5 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g.

): Linux staging-controller-002 4.4.0-139-generic -Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: ansible Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/cc seems that it's the graph in node authz is not ready to hande requests. probably respecting infomer's func can make it initialized gracefully. while the log you provides didn't show why the apiserver's crushed. could you provide more detail?
if this occurred right at server start, it is expected the graph will take a short amount of time to populate
If you tell me what more info you need, I'll be glad to provide you with it, maybe the apiserver flags used? Also, I restart the one crash-dumping, one of the other two will start outputting the same dump. Looking back at the logs, this seems to have started when I moved from 1.10.3 to 1.12.1 (now 1.12.2), I just hadn't realized it until recently as we had a big cluster wide problem yesterday and had to investigate some more. But it is still outputting this same dump non-stop even though it seems to be working at the moment and that API server does respond to kubectl commands without issues. Any specific info you'd like me to get?
from the stack trace, sadly we can't find out which exact line trigger that nil pointer panic. the behavior of apiserver could be unstable on the fresh start (due to rbac authz, crd-apiservice registration, etc). are we planing a graceful start mechanism for the server?
the apiserver endpoint is used to determine if the server is ready to accept traffic, and will not return a 200 response until those startup tasks have completed
/healthz does respond (200), while still crash-dump-looping. EDIT: anything I can test specifically to help differentiate between RBAC authz, crd-apiservice reg, etc?
right, I was saying that is the mechanism for graceful startup. unfortunately, the way golang works, if you propagate a panic (), you lose all stack information from it by default. Some of our panic propagation logs or preserves that, but not all.

I'll open a fix to preserve that information. we can see the request that errored, though: can you see if the same resource (autoscaling/v2beta1/horizontalpodautoscalers) is what causes the panic each time? if so, can you check that resource per namespace to isolate a particular problematic object? those are not related to the panic, as far as I can tell
Still reading, but: Did you skip 1.11? Skipping releases isn't actually supported.
Right now, having been up for a while, it alternates almost perfectly (E114 sometime coming twice in a row) between E114 and I114, basically once per second. These two errors correspond to the one you quoted (HPA) & the other to the HTTP2 panic. The cluster was upgraded from 1.10.3 to 1.12.1 (to 1.12.2) and from what I could gather last week, this started after the upgrade to 1.12.x branch but we didn't realize it until we had to investigate a cluster issue 3 weeks after the upgrade.
Yes, I did, I didn't know that... should I roll back to 1.11 and than back again to 1.12? Or what are the effects and what can be done?
There's at least two things going wrong, one is: the other is whatever is actually crashing. At this point I'm not sure if rolling back to 1.11 would make things worse or better, if your cluster is functioning I probably wouldn't do; if your cluster is nonfunctional and you need it fixed then I might roll the dice. I think we should be able to get a fix for the first thing out soon, but it'll be a while before you'd be able to deploy it unless you're willing to build your own apiserver & run it.
So, any way to diagnose what should have been updated in, I guess, etcd? I mean, is this similar to a schema change that is missing some steps? Thanks!
callstack logging fixed in and
Any chance this will also get merged in 1.12.3 so I could get a meaningful crash dump with the fix?

:)
sure, this will get into 1.12.3 via
So here is the new dump I've got under 1.12.3: Does it make more sense to anyone and is it pointing to the reason for the stack dump?
apologise, it didn't get into 1.12.3. (i thought it would). if you don't feel like waiting until next release, how about building ur own apiserver from release-1.12 HEAD as suggests? (run under this project)
Oh well, I can wait for 1.12.4... (Building it didn't work out and I'd have to build a custom docker, etc. I'll wait for the next release...)
We have the same issue and we reproduced with new kubernetes api-server Stacktrace:
thx for providing the clue! fixed via