Question: }) } <ins> // TimeoutHandler returns an http.Handler that runs h with a timeout // determined by timeoutFunc. The new http.Handler calls h.ServeHTTP to handle // each request, but if a call runs for longer than its time limit, the // handler responds with a 503 Service Unavailable error and the message // provided. (If msg is empty, a suitable default message with be sent.) After // the handler times out, writes by h to its http.ResponseWriter will return // http.ErrHandlerTimeout. If timeoutFunc returns a nil timeout channel, no // timeout will be enforced. func TimeoutHandler(h http.Handler, timeoutFunc func(*http.Request) (timeout <-chan time.Time, msg string)) http.Handler { return &timeoutHandler{h, timeoutFunc} } type timeoutHandler struct { handler http.Handler timeout func(*http.Request) (<-chan time.Time, string) } func (t *timeoutHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { after, msg := t.timeout(r) if after == nil { t.handler.ServeHTTP(w, r) return } done := make(chan struct{}, 1) tw := newTimeoutWriter(w) go func() { t.handler.ServeHTTP(tw, r) done <- struct{}{} }() select { case <-done: return case <-after: tw.timeout(msg) } } type timeoutWriter interface { http.ResponseWriter timeout(string) } func newTimeoutWriter(w http.ResponseWriter) timeoutWriter { base := &baseTimeoutWriter{w: w} _, notifiable := w.(http.CloseNotifier) _, hijackable := w.(http.Hijacker) switch { case notifiable && hijackable: return &closeHijackTimeoutWriter{base} case notifiable: return &closeTimeoutWriter{base} case hijackable: return &hijackTimeoutWriter{base} default: return base } } type baseTimeoutWriter struct { w http.ResponseWriter mu          sync.Mutex timedOut    bool wroteHeader bool hijacked    bool } func (tw *baseTimeoutWriter) Header() http.Header { return tw.w.Header() } func (tw *baseTimeoutWriter) Write(p []byte) (int, error) { tw.mu.Lock() defer tw.mu.Unlock() tw.wroteHeader = true if tw.hijacked { return 0, http.ErrHijacked } if tw.timedOut { return 0, http.ErrHandlerTimeout } return tw.w.Write(p) } func (tw *baseTimeoutWriter) WriteHeader(code int) { tw.mu.Lock() defer tw.mu.Unlock() if tw.timedOut || tw.wroteHeader || tw.hijacked { return } tw.wroteHeader = true tw.w.WriteHeader(code) } func (tw *baseTimeoutWriter) timeout(msg string) { tw.mu.Lock() defer tw.mu.Unlock() if !tw.wroteHeader && !tw.hijacked { tw.w.WriteHeader(http.StatusGatewayTimeout) if msg != "" { tw.w.Write([]byte(msg)) } else { enc := json.NewEncoder(tw.w) enc.Encode(errors.NewServerTimeout("", "", 0)) } } tw.timedOut = true } func (tw *baseTimeoutWriter) closeNotify() <-chan bool { return tw.w.(http.CloseNotifier).CloseNotify() } func (tw *baseTimeoutWriter) hijack() (net.Conn, *bufio.ReadWriter, error) { tw.mu.Lock() defer tw.mu.Unlock() if tw.timedOut { return nil, nil, http.ErrHandlerTimeout } conn, rw, err := tw.w.(http.Hijacker).Hijack() if err == nil { tw.hijacked = true } return conn, rw, err } type closeTimeoutWriter struct { *baseTimeoutWriter } func (tw *closeTimeoutWriter) CloseNotify() <-chan bool { return tw.closeNotify() } type hijackTimeoutWriter struct { *baseTimeoutWriter } func (tw *hijackTimeoutWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { return tw.hijack() } type closeHijackTimeoutWriter struct { *baseTimeoutWriter } func (tw *closeHijackTimeoutWriter) CloseNotify() <-chan bool { return tw.closeNotify() } func (tw *closeHijackTimeoutWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { return tw.hijack() } </ins> // TODO: use restful.CrossOriginResourceSharing // Simple CORS implementation that wraps an http Handler // For a more detailed implementation use https://github.com/martini-contrib/cors

I noticed that if I'm following a pod log that is being actively written to it will timeout after 5 minutes. However if the pod is not writing logs the request will not timeout. In that case the request will remain open seemingly indefinitely. I can change the behavior by modifying this line of code: To make it happen faster you can set the value lower. I've been testing with 1 minute. Steps to reproduce: cluster the nginx pod in another terminal generate log output by hitting the pod: (you'll need to change the ip). the logs: In this case whenever the WriteTimeout is reached curl will receive the zero sized chunk and the connection will be closed. If you leave out step 3 curl will continue seemingly indefinitely. This behavior definitely seems odd. Is the usage of WriteTimeout correct here? A quick search shows that others have been confused by the intention of WriteTimeout which is why I ask.
is this same as ?
From discussions on the OpenShift side it sounds somewhat related however what we're talking about is a reasonable (but arbitrary) timeout set. Users of are subjected to this timeout with no good way to pick up where they left off to avoid redownloading potentially large quantities of logs.
To me, it would make sense if the api endpoint had argument which would fetch logs from line (it's easy to track on client on which line we're at), exactly what does: I'm assuming that kubelet gets logs using docker api, unfortunately there is no such thing in docker's api:
I think users that want to do this should get the logs from elastic search instead of from kubectl.
I can't say I disagree about needing something like elastic search in a real world deployment. I guess part of the weirdness is that clients of the logging API rarely know how much is going to be pulled down before they make the request. I guess the answer now if you're following the logs is "5 minutes worth". I know this isn't how it's implemented today but in my mind following the logs should behave more like or . Meaning, when you state you want to follow you don't fetch the entire log file you just get wherever it happens to be (and maybe a little more for context). If you want the whole file you don't pass in .
Any update? This is really annoying and I'm thinking of patching my k8s.

I think the default API timeout has been lengthened to 60 minutes Defined: Used for API server: It looks like the kubelet timeout is still 5 minutes, so that would gate log streaming to 5 minutes:
More nuances in Looks like the API http server times out after 60 minutes: Watches time out between 5 and 10 minutes: Kubelet http server still times out after 5 minutes
Which means that getting logs timeouts after 5 minutes?
yeah
did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
nm, just merged the pr. did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
we could easily bump up the kubelet timeout. The handlers on the kubelet don't have their own built in timeouts (like I did for watch), so a potential downside is long hanging /stats requests could hog connections, and depending on why they're hanging, continue consuming kubelet/cpu/cadvisor time. had a suggestion previously about using elasticsearch, are you running with a cluster that doesn't have the addons (, if you set it up with kube-up it should)?
IMO there's no reason not to make kubelet server's timeout really long, at least for these requests.
I'm trying to put in a fix to increase the kubelet timeouts to 1 hour to match the master. Unfortunately adding proper timeout handling for these cases may not make 1.0, but hopefully this can make it less annoying.


Question: ) flagSet.StringVar(featureGatesString, "feature-gates", *featureGatesString, "A set of key=value pairs that describe feature gates for various features. "+ "Options are:n"+strings.Join(features.KnownFeatures(&features.InitFeatureGates), "n")) <ins> flagSet.Var(utilflag.NewMapStringString(&cfg.APIServerExtraArgs), "apiserver-extra-args", "A set of extra flags to pass to the API Server or override default ones in form of <flagname>=<value>") flagSet.Var(utilflag.NewMapStringString(&cfg.ControllerManagerExtraArgs), "controller-manager-extra-args", "A set of extra flags to pass to the Controller Manager or override default ones in form of <flagname>=<value>") flagSet.Var(utilflag.NewMapStringString(&cfg.SchedulerExtraArgs), "scheduler-extra-args", "A set of extra flags to pass to the Scheduler or override default ones in form of <flagname>=<value>") </ins> } // AddInitOtherFlags adds init flags that are not bound to a configuration file to the given flagset

/kind feature What happened: When calling , there is no way to override default flag values for the different components of the controlplane. This is surprising, as in the config structure (and in controlplane phase codebase itself), there are explicit settings for doing that (APIServerExtraArgs, ControllerManagerExtraArgs, SchedulerExtraArgs). What you expected to happen: Those config fields should be exposed as flags on and How to reproduce it (as minimally and precisely as possible): Try to deploy K8s using Kubeadm with a custom Admission Control setting (I'll follow up on this issue with a PR)
btw, I am not sure which sig owns kubeadm.
it's sig-cluster-lifecycle.
/sig sig-cluster-lifecycle
/sig cluster-lifecycle


Question: // CleanScope returns the scope of the request. func CleanScope(requestInfo *request.RequestInfo) string { <del> if requestInfo.Name != "" { </del> <ins> if requestInfo.Name != "" || requestInfo.Verb == "create" { </ins> return "resource" } if requestInfo.Namespace != "" {

I've observed, that some metrics has and labels. (create) requests are "single resource", therefor they should be classified as . Def: customresourcehandler uses , which is based on the RequstInfo only: the RequestInfo is based only on the requests: method and url example: e.g: Since this RequestInfo has Name and Namespace empty, it is classified as . In a similar way it could be explained why: There's a mismatch in scope classification between built-in and custom resources: The key difference - the check if either "name" was provided OR it is a : Where the metrics.CleanScope ONLY checks, if name is not-empty. Tested: = v1.20 There were other scope related issues <1.20 and the distinction is not obvious. <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: // start the kubelet server if *enableServer { go util.Forever(func() { <del> kubelet.ListenAndServeKubeletServer(k, cfg.Channel("http"), http.DefaultServeMux, *address, *port) </del> <ins> kubelet.ListenAndServeKubeletServer(k, cfg.Channel("http"), *address, *port) </ins> }, 0) }

The Kubelet REST end-point has turned into a complicated switch-case code block that is harder to maintain overtime. The Kubelet should move to a http.ServeMux pattern and register handler functions to a pattern.
I am working on a PR to submit in next couple of days that will make this change, so feel free to assign to me if possible.
Tagging because I asked for this in one of his PRs
This one is also under my radar. If no one had time, feel free to assign it to me too.
I have a branch in progress. Feel free to offer feedback on it when I submit. Sent from my iPhone


Question: return } <ins> func (f *FakeDockerClient) AssertUnorderedCalls(calls []string) (err error) { f.Lock() defer f.Unlock() var actual, expected []string copy(actual, calls) copy(expected, f.called) sort.StringSlice(actual).Sort() sort.StringSlice(expected).Sort() if !reflect.DeepEqual(actual, expected) { err = fmt.Errorf("expected(sorted) %#v, got(sorted) %#v", expected, actual) } return } </ins> // ListContainers is a test-spy implementation of DockerInterface.ListContainers. // It adds an entry "list" to the internal method call record. func (f *FakeDockerClient) ListContainers(options docker.ListContainersOptions) ([]docker.APIContainers, error) {

Seeing random failures.
Looks like lately refactory makes docker API requests are not deterministic. Will look into it.
Any update? It seems to be failing a lot.
I just looked into this and found that is the only test that there are two pods in the which is fed to I think podworker was syncing the two pods in parallel, this causes non-deterministic behaviour I have tried , no errors. , error happens continuously. I can try to fix this :)
Yes, that is what I believe here, but never have time working on it. Thanks for looking into this, and re-assign it to you.
I saw your old PR for this, I think we still need to keep the parallel sync here. Just need to create a for this case :)


Question: handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc) handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup) handler = genericapifilters.WithRequestInfo(handler, server.NewRequestInfoResolver(c)) <ins> handler = genericapifilters.WithCacheControl(handler) </ins> handler = genericfilters.WithPanicRecovery(handler) return handler

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: cfg.API.AdvertiseAddresses = []string{ip.String()} } <ins> if !skipPreFlight { fmt.Println("Running pre-flight checks") err := preflight.RunInitMasterChecks(cfg) if err != nil { return nil, &preflight.PreFlightError{Msg: err.Error()} } } else { fmt.Println("Skipping pre-flight checks") } </ins> // TODO(phase1+) create a custom flag if cfg.CloudProvider != "" { if cloudprovider.IsCloudProvider(cfg.CloudProvider) {

Environment: VMware CentOS Linux release 7.2.1511 (Core) kernel: 3.10.0-327.36.1.el7.x8664 Installed packages: kubeadm.x8664 1.5.0-0.alpha.0.1534.gcf7301f kubectl.x8664 1.4.0-0 kubelet.x8664 1.4.0-0 kubernetes-cni.x8664 0.3.0.1-0. docker-engine.x8664 1.11.2- docker-engine- 1.12.1- What happened: kubeadm init hangs indefintely after: kubeadm init <master/tokensgenerated token: "." <master/pkicreated keys and certificates in "/etc/kubernetes/pki" <util/kubeconfigcreated "" <util/kubeconfigcreated "" <master/apiclientcreated API client configuration <master/apiclientcreated API client, waiting for the control plane to become readykubeadm init What you expected to happen: Master node initialize Anything else do we need to know: Only the kubeadm and kublet processes are running. When I do docker ps no containers are running. If i type docker images the only image that has been downloaded are: I've also tried: kubeadm init --api-advertise-addresses but with the same outcome (followed the clean-up at ) Im behind a firewall so I'm using proxy settings for docker. I have confirmed I can reach images on docker hub and google cloud registry
will download the required binaries viz etcd. When you are behind firewall kubeadm will hang/wait forever to download the binaries. I dont know whether it is addressed correctly in kubeadm implementation. I am also facing the same issue and waiting for the resolution.
Thx, I suspected it to be some firewall issue. Do you know if there's a way to add httpsproxy & httpproxy settings to kubeadm?
Kubeadm doesn't seem to pick proxy environment variables. I am looking at recompiling with proxy support may work. But I dont know how to do that. Waiting for updates about proxy
This issue same as Lets join there. The solution mentioned there may work.

You can try kubeadm with patch applied from PR : It works at least in my environment behind restrictive proxy.
thanks, I will look into it


Question: "STANDARD_D4":            32, "STANDARD_D48AS_V4":      32, "STANDARD_D48A_V4":       32, <ins> "STANDARD_D48DS_V4":      32, "STANDARD_D48D_V4":       32, </ins> "STANDARD_D48S_V3":       32, "STANDARD_D48_V3":        32, "STANDARD_D4AS_V4":       8, "STANDARD_D4A_V4":        8, <ins> "STANDARD_D4DS_V4":       8, "STANDARD_D4D_V4":        8, </ins> "STANDARD_D4S_V3":        8, "STANDARD_D4_V2":         32, "STANDARD_D4_V2_PROMO":   32,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: set -o nounset set -o pipefail <ins> echo "Note: This assumes that types_swagger_doc_generated.go has been updated for all API group versions. Please run hack/update-generated-swagger-docs.sh to ensure that." </ins> KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. SWAGGER_ROOT_DIR="${KUBE_ROOT}/api/swagger-spec" source "${KUBE_ROOT}/hack/lib/init.sh"

Ref: verify-swagger-spec is always passing on both travis and jenkins even when the swagger spec is not updated. It fails when run locally if the spec is not updated. Need to figure out why does it not fail on travis and jenkins.
It is failing now: https://travis- All I did was add some echo statements :)
Sent to fix this


Question: numErrs: 1, }, { <ins> name: "invalid publicIPs localhost", makeSvc: func(s *api.Service) { s.Spec.PublicIPs = []string{"127.0.0.1"} }, numErrs: 1, }, { name: "invalid publicIPs", makeSvc: func(s *api.Service) { s.Spec.PublicIPs = []string{"0.0.0.0"} }, numErrs: 1, }, { name: "valid publicIPs host", makeSvc: func(s *api.Service) { s.Spec.PublicIPs = []string{"myhost.mydomain"} }, numErrs: 0, }, { </ins> name: "nil selector", makeSvc: func(s *api.Service) { s.Spec.Selector = nil

When spinning up VMs w/ kube, I attempted to use but this fails. ironically, 127.0.0.1 succeeds (internally you can then access a kube service via , for example, but you cannot do this if you try to bind to 0.0.0.0. binding works to 127.0.0.1, but is a horrible idea (since this IP wont be reachable by the outside world), so in that case kube should print a clear message on why this is the case and fail. binding to 0.0.0.0 fails right out of the bat, so in this case, also, kube should fail immediately. To be clear, im suggesting that these should fail at the service submission layer. Possibly a module for validating logic of submitted services/pods , which extends syntactical validation is in order if one doesn't already exist?
Please also see
Since this is very similar to I'm gonna take this.


Question: EXTRA_DOCKER_OPTS="${EXTRA_DOCKER_OPTS:-}" INSTANCE_PREFIX="${KUBE_AWS_INSTANCE_PREFIX:-e2e-test-${USER}}" <ins> CONFIG_CONTEXT="${KUBE_CONFIG_CONTEXT:-aws_${INSTANCE_PREFIX}}" </ins> CLUSTER_ID=${INSTANCE_PREFIX} AWS_SSH_KEY=${AWS_SSH_KEY:-$HOME/.ssh/kube_aws_rsa} IAM_PROFILE_MASTER="kubernetes-master"

I've run to create a cluster, which resulted in a cluster and user being created in kube-config. Now I want to run again with different parameters, to create a second cluster in a different AWS account. I would like to use a different name for the cluster and user for this cluster. From what I can see from the , there seems to be no way to completely override the name used. I understand that I can use a separate kube-config file, but I don't want to do that. I want to use the same kube-config file, with multiple clusters and users, all handled by .
Manually overwriting file to set CONTEXT to a different value works to achieve the effect that I want. This value should be allowed to be set externally so that we don't have to resort to manually editing the file for each cluster.
Fixed in


Question: Note that the delete command does NOT do resource version checks, so if someone submits an update to a resource right when you submit a delete, their update will be lost along with the <del> rest of the resource.`)) </del> <ins> rest of the resource. After a CustomResourceDefinition is deleted, invalidation of discovery cache may take up to 10 minutes. If you don't want to wait, you might want to run "kubectl api-resources" to refresh the discovery cache.`)) </ins> deleteExample = templates.Examples(i18n.T(` # Delete a pod using the type and name specified in pod.json

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Changing the scope field of a CRD ist not possible since it's immutable. But even after deleting and re-creating the CRD with changed scope it doesn't let me create resources of the CRD's kind. Re-creating CRD with changed scope should just work. Here is a full example that in the end leads to the error when trying to create a resource after re-creating the CRD. Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: vSphere 7.0 OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1160.42.2.el7.x86_64
/sig api-machinery
/assign /triage accepted
This appears to be working as intended. The example custom resource you use: It Cluster scoped because it has no namespace. But once the CRD is namespace scoped, you should not be able to create cluster scoped custom resources of that type. If instead you add a namespace to Example, it should be created, e.g.:
if the above seems to have answered your question - please feel free to close the issue :) Thank you!
kubectl will automatically use the namespace so this is definitely not the issue here. I can reproduce this issue with my above script in a fresh (v1.22.2) cluster. You can add the namespace in the last manifest, it makes no difference. From my understanding this really seems to be a bug.
Yes, that's true. Good catch. Okay, we need to root cause this.
I tried using curl to create the custom resource directly against the API and got the same result: So the problem seems to be in the API server (not in kubectl).
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: d := schemaCoercingDecoder{delegate: ret.StorageConfig.Codec, validator: unstructuredSchemaCoercer{ // drop invalid fields while decoding old CRs (before we haven't had any ObjectMeta validation) dropInvalidMetadata:   true, <ins> repairGeneration:      true, </ins> structuralSchemas:     t.structuralSchemas, structuralSchemaGK:    t.structuralSchemaGK, preserveUnknownFields: t.preserveUnknownFields,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: func(ctrl *PersistentVolumeController, reactor *pvtesting.VolumeReactor, test controllerTest) error { volume := ctrl.volumes.store.List()[0].(*v1.PersistentVolume) reactor.DeleteVolumeEvent(volume) <del> for len(ctrl.volumes.store.ListKeys()) > 0 { time.Sleep(10 * time.Millisecond) </del> <ins> err := wait.Poll(10*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { return len(ctrl.volumes.store.ListKeys()) == 0, nil }) if err != nil { return err </ins> } // trying to remove the claim as well obj := ctrl.claims.List()[0] claim := obj.(*v1.PersistentVolumeClaim) reactor.DeleteClaimEvent(claim) // wait until claim is cleared from cache, i.e., deleteClaim is called <del> for len(ctrl.claims.ListKeys()) > 0 { time.Sleep(10 * time.Millisecond) </del> <ins> err = wait.Poll(10*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { return len(ctrl.claims.ListKeys()) == 0, nil }) if err != nil { return err </ins> } // make sure operation timestamp cache is empty if ctrl.operationTimestamps.Has("volume5-7") {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are some places in that wait for conditions . We should update all the places to set something like /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: There are some places in that wait for conditions . We should update all the places to set something like /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /remove-kind bug
I can try to fix this :) /assign
/sig storage


Question: podWarnings        []string podWarningsToCount = make(map[string]podCount) <ins> prioritizedPods    = a.prioritizePods(pods) </ins> ) <del> totalPods := len(pods) if len(pods) > a.namespaceMaxPodsToCheck { pods = pods[0:a.namespaceMaxPodsToCheck] </del> <ins> totalPods := len(prioritizedPods) if len(prioritizedPods) > a.namespaceMaxPodsToCheck { prioritizedPods = prioritizedPods[0:a.namespaceMaxPodsToCheck] </ins> } <del> checkedPods := len(pods) for i, pod := range pods { // short-circuit on exempt runtimeclass if a.exemptRuntimeClass(pod.Spec.RuntimeClassName) { continue } </del> <ins> checkedPods := len(prioritizedPods) for i, pod := range prioritizedPods { </ins> r := policy.AggregateCheckResults(a.Evaluator.EvaluatePod(enforce, &pod.ObjectMeta, &pod.Spec)) if !r.Allowed { warning := r.ForbiddenReason()

When updating policy labels on a namespace, existing pods are evaluated to see if they are compatible with the new policy. If there are too many pods, or if evaluation takes too long, we short circuit and warn that not all pods were checked. For that reason, we want to prioritize checking a variety of pods before checking essentially identical pods, so that unique warnings can be surfaced before reaching the pod count cutoff or evaluation timeout. Pods created by the same controller are most likely to be identical (e.g. replicas from a replicaset or job, copies of pods from a statefulset or daemonset, etc) cc
This is an interesting fix I'd like to take this up Can I take this up? I am fairly new to Kubernetes source code so might need some help
I am guessing this is where the logic you mentioned should be implemented right?
thanks for the interest; this is targeted at the beta version... right now, effort is going toward the alpha-blocking work for 1.22
Should I wait for the code freeze before working on this? Sorry for lame questions
If you want to start looking at it, that's fine, but it's possible the code in the ValidateNamespace function will change before we're ready to start merging 1.23 work, and there isn't really reviewer bandwidth to take a look at work-in-progress stuff for next release yet.
Sure, I'll just for code freeze before I start with implementation By the time I'll just assign this to myself /assign
Does it make sense to take it up now? I think we already have a couple of RC builds after the code freeze
sure, it's fine to start work in a branch in your fork and have a WIP PR open, even before master reopens for 1.23 work
Awesome, I'll start working on it Thank you
Hello! Are you still working on this? If you don't have time, I can take this issue. If not, maybe I can give some help
Since this was not an urgent issue I was working on some other issues last week I have started on this, Will be raising a PR maybe today or tomorrow I Will ask you if I get stuck Thank you


Question: } func (t *Tester) testWatchLabels(obj runtime.Object, emitFn EmitFunc, labelsPass, labelsFail []labels.Set, actions []string) { <del> ctx := t.TestContext() </del> <ins> ctx := t.TestContext().(context.Context) </ins> for _, label := range labelsPass { for _, action := range actions { options := &api.ListOptions{LabelSelector: label.AsSelector(), ResourceVersion: "1"} <ins> ctx = context.WithValue(context.WithValue(ctx, "label", label), "action", action) </ins> watcher, err := t.storage.(rest.Watcher).Watch(ctx, options) if err != nil { t.Errorf("unexpected error: %v", err)

https://k8s- Failed: TestWatch {etcd} Previous issues for this test:
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
/cc
are there any known issues on the clientv3 prior to 3.0.8? Just wondering if we need a rev bump.
I can never repro this locally, and the leaks have been fixed.
https://k8s- Failed: TestWatch {etcd}
So it's failing in different spots, but replicasets is the most common occurrence.
xref:
ok, we have a bead on the issue. It will not affect the released bits, but it does affect the UTs.
PR in flight.
What's the default logging level of unit test?
If we looked at the test, they are sort of wrong themselves. For example, It's doing another action on the same key, but it's watching at version "1" again. From my understanding, if we have action "create, delete", I want to watch two actions one by one. But it's watching at "1", now we just watch "create", then another same "create".
Even worse, it could be hiding problems. For example, if there is some event that happened in previous suites, then it will happen to next suites without differentiating and checking the actual values from ResultChan(). In such case, a wrong test might pass.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
- if was supposed to help, it didn't - it is still failing.
has some ideas... and I'm digging too.
https://k8s- Failed: TestWatch {etcd}
Here's my understanding of the scenario: What's happening? A lot of get requests are sent due to prev-value. Note that they are quorum get. Those get request are cancelled due to the wrong logic in tests. What's happening inside etcd is that etcd server is still processing them and trying to send them until grpc fails. These get requests could block further events from being processed.

It gets even worse under heavy loaded environment. What's the problem in the unit test itself? Their logic are wrong themselves. See my previous comments. We need to fix the tests. What's on etcd? I need to stress test this behavior. Then I will coordinate with etcd team to see potentially what could be improved/fixed.
There is just a single etcd instance - so whether this is quorum or not doesn't matter here. Actually - it is a valid usecase that user cancel the request (if I'm not missing anything subtle here). ^
Quorum get affects processing events. The problem is not the cancelling. See my previous comments on the test itself. If we expect to test cancelling requests, then we should have another set of tests for that, and have different checking workflow.
I think the tests watching a "1" expect a new etcd instance each time. Otherwise they would be horribly broken.
https://k8s- Failed: TestWatch {etcd}
I'm still digging into "how" this happens... we fixed a bunch of issues. It's clear the watch is timing out.. sometimes. Every single test fires up it's own client/server pair, each of which is also properly cleaned... now.
share your repro steps?
when digging through latest output across the tests e.g. I'm always seeing a triplet of: But tests always pass, still digging...
This is the problem caused by watching "1". It keeps getting previous events. Meanwhile, watching "1" has some other latency issue. I had done some benchmark that does watch from 1 and latency could spike up to 5 second. I need more time to do more testing on this. But the tests themselves are problematic. We should fix them first. If it doesn't fix the problem, we will try to reproduce it then.
I was assuming your previous comment says that you have changed/fixed the tests and you are still reproducing the problem?
Why is watching from "1" problematic? There shouldn't be any previous events, since we start etcd fresh for every test. , Hongchao Deng wrote:
The tests are problematic. Inside each test there are many test suites not getting a fresh etcd.

Watching "1" will force etcd to query internal database for past events, potentially causing some latency issue on compacted env. With all these said, it's all theories. We still need to do more logging and testing around it to find out real cause.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
for searchability
I see what is saying now re: rewatch @ 1. The tests areas that are failing are in loops that are constantly re-issuing a watch at 1, even though b/c it's in a loop emitting it should not be 1, tests work fine when simply removing entirely. So I'll put together a PR that ups the timeout + removed the ListOption. Although I do think the negative tests should probably be re-written to interleave with the positive tests, to eliminate the time suck.
Thanks. I am very much looking forward to that PR. ;)
Do you have an ETA for the PR to help us figure out if this should be moved to flaky as a stop-gap?
we're still debugging, what doesn't help is that these failures only seem to occur on GCE and no one care repro them outside of that environment.
Frustrating. Are you able to reliable reproduce using the "run until failure" test option?

Merged
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
Still occurring xref:
Yes, just happened 3x in a row here:
Do you think we can create a new etcd instance every time? As mentioned in
won't fix the problem. I do have another workaround for this until etcd fix comes in.
is on


Question: <ins> apiVersion: v1 kind: Service metadata: name: flocker-ghost labels: app: flocker-ghost spec: ports: # the port that this service should serve on - port: 80 targetPort: 80 selector: app: flocker-ghost --- apiVersion: v1 kind: ReplicationController metadata: name: flocker-ghost # these labels can be applied automatically # from the labels in the pod template if not set labels: purpose: demo spec: replicas: 1 template: metadata: labels: app: flocker-ghost spec: containers: - name: flocker-ghost image: ghost:0.7.1 env: - name: GET_HOSTS_FROM value: dns ports: - containerPort: 2368 hostPort: 80 protocol: TCP volumeMounts: # name must match the volume name below - name: ghost-data mountPath: "/var/lib/ghost" volumes: - name: ghost-data flocker: datasetName: my-flocker-vol </ins>

We found that some users coming to the document were getting a little confused. It should include more detail about where to install Flocker services, how to set environment variables, that you can create volumes now with Docker CLI and give more detail in example including how to use it with a spec that includes a replication controller.


Question: "github.com/golang/glog" "k8s.io/kubernetes/pkg/cloudprovider" "k8s.io/kubernetes/pkg/types" <ins> "k8s.io/kubernetes/pkg/volume" </ins> ) const ( <del> maxLUN = 64 // max number of LUNs per VM </del> <ins> maxLUN               = 64 // max number of LUNs per VM errLeaseFailed       = "AcquireDiskLeaseFailed" errLeaseIDMissing    = "LeaseIdMissing" errContainerNotFound = "ContainerNotFound" </ins> ) // AttachDisk attaches a vhd to vm

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: Ansible What happened: PVC keeps pending with a new created storage account What you expected to happen: Storage class can find storage account and then create a dynamic pv for pvc to bind How to reproduce it (as minimally and precisely as possible): a new storage account without create any container a storage class as below a pvc annotating this storage class pvc status is pending Anything else do we need to know: After I create a new container "vhds" in storage account of "qewestus", storage class works and pvc is bound
VHD container is a rerequiste. Will update the doc.
.. I was about to open this exact bug. Shouldn't the provisoner create the vhds container ?
yes, there is a todo for this case (also missing storage account)
Another minor issue is, new volumes take a lot of time (tens of mins) to be ext4 formatted. This causes the pod to timeout and fail upon first use! Any way to make this better
fix is in
Can anyone share how to create the vhds container? For those that don't yet have this fix
Easiest is through the portal, browse to your storage account and create a container called
Thats great thanks for the quick response. The automator in me now wants to see if I can automate the creation of that container through powershell ;)
Does the container have to be of type 'Container' ? IE Full public read access??

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Linux Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: hack/local-up- What happened: storage class pvc annotated with this storage class a pod consume this pvc read/write function with the mounted path pod/pvc the pv status which is "Failed" What you expected to happen: PV should not have "Failed" status before deleted How to reproduce it (as minimally and precisely as possible): Always Anything else do we need to know:
cc
Per message The disk is still being attached. As a result, PV cannot be deleted.
This condition is handled by the PV framework. See However the volume plugin must return which it looks like the Azure volume plugin is not doing.
please help remove label, kubectl can't do anything with this.


Question: file_check_frequency=10s pod_cidr=10.100.0.0/24 log_level=4 <del> start_kubelet --kubeconfig "${KUBELET_KUBECONFIG_DIR}/kubelet.kubeconfig"  </del> <ins> start_kubelet --kubeconfig ${KUBELET_KUBECONFIG}  </ins> --volume-stats-agg-period $volume_stats_agg_period  --allow-privileged=$allow_privileged  --serialize-image-pulls=$serialize_image_pulls

/kind bug What happened: From doest not find kubeconfig file What you expected to happen: To find kubeconfig file How to reproduce it (as minimally and precisely as possible): from conformance file Anything else we need to know?: Line 185 : Starting kubelet with args But does not exist. Using from line 80 seems to work properly Environment: Kubernetes version (use ):Cloud provider or hardware configuration: hardware configuration OS (e.g. from /etc/os-release): Fedora 28 Server Kernel (e.g. ): Linux 4.17.12-200.fc28.x86_64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area conformance


Question: } podID := u.Query().Get("podID") podUUID := u.Query().Get("UUID") <ins> podNamespace := u.Query().Get("podNamespace") </ins> if len(podID) == 0 { w.WriteHeader(http.StatusBadRequest) http.Error(w, "Missing 'podID=' query entry.", http.StatusBadRequest) return } <ins> if len(podNamespace) == 0 { w.WriteHeader(http.StatusBadRequest) http.Error(w, "Missing 'podNamespace=' query entry.", http.StatusBadRequest) return } </ins> // TODO: backwards compatibility with existing API, needs API change podFullName := GetPodFullName(&api.BoundPod{ TypeMeta: api.TypeMeta{ <del> ID: podID, // TODO: I am broken Namespace:   api.NamespaceDefault, </del> <ins> ID:          podID, Namespace:   podNamespace, </ins> Annotations: map[string]string{ConfigSourceAnnotationKey: "etcd"}, }, })

The podInfo call has no way to accept namespace, which means the apiserver can't fetch pod info. Probably need to make namespace a required path parameter.
this is probably you :)
- is this something you see when working off ??
- ok so this is pod_cache, will review
If you don't time, I can pick up this one.
I got it.
Per discussion with , will deliver a pull after is merged.


Question: // ScaleApplyConfiguration constructs an declarative configuration of the Scale type for use with // apply. func Scale() *ScaleApplyConfiguration { <del> return &ScaleApplyConfiguration{} </del> <ins> b := &ScaleApplyConfiguration{} b.WithKind("Scale") b.WithAPIVersion("autoscaling/v1") return b </ins> } // WithKind sets the Kind field in the declarative configuration to the given value

Most of the FooApplyConfiguration constructors pre-fill the Kind and APIVersion fields to make using them smooth. But () is missing this, making patterns like this fail: Currently this code fails with (server-side) error . The constructor should set the minimum required fields for the resulting object to be usable with calls, similar to how e.g. the `D Create a deployment (however) Create a clientset, and get the Deployments client from it Try to call deployments.ApplyScale using a ScaleApplyConfiguration created by the Scale() constructor I can see that the presence or absence of these metadata fields is driven by some code generation, but I wasn't able to follow through to what configures that code generation. I assume it's something in the schemas somewhere, but that wasn't clear to me. <details</details<detailsminikube v1.25.1 </detailsN/A N/A N/A N/A
/sig api-machinery
/assign /triage accepted
/assign


Question: gomega.Expect(e2epod.WaitForPodToDisappear(c, ns, name, labels.Everything(), pollInterval, pollTimeout)).To(gomega.Succeed()) ginkgo.By("Delete the config map") c.CoreV1().ConfigMaps(ns).Delete(context.TODO(), configName, metav1.DeleteOptions{}) <del> ginkgo.By("Delete the service account") c.CoreV1().ServiceAccounts(ns).Delete(context.TODO(), name, metav1.DeleteOptions{}) ginkgo.By("Delete the clusterRoleBinding") c.RbacV1().ClusterRoleBindings().Delete(context.TODO(), name, metav1.DeleteOptions{}) </del> ginkgo.By("Clean up the events") gomega.Expect(c.CoreV1().Events(eventNamespace).DeleteCollection(context.TODO(), *metav1.NewDeleteOptions(0), eventListOptions)).To(gomega.Succeed()) ginkgo.By("Clean up the node condition")

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: import ( "testing" <ins> "time" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> certificates "k8s.io/kubernetes/pkg/apis/certificates/v1beta1" "k8s.io/kubernetes/pkg/client/clientset_generated/clientset/fake" informers "k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions"

Seen in happy to send a fix but didn't know if this might have been address with your current refactors. Output:
This looks new. I'd happily review a fix. Thanks.
Dupe of


Question: }, }, { <ins> name: "terminal phase from previous status must remain terminal, restartAlways", pod: &v1.Pod{ Spec: desiredState, Status: v1.PodStatus{ Phase: v1.PodRunning, ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, currentStatus: &kubecontainer.PodStatus{}, previousStatus: v1.PodStatus{ Phase: v1.PodSucceeded, ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, // Reason and message should be preserved Reason:  "Test", Message: "test", }, expected: v1.PodStatus{ Phase:    v1.PodSucceeded, HostIP:   "127.0.0.1", QOSClass: v1.PodQOSBestEffort, Conditions: []v1.PodCondition{ {Type: v1.PodInitialized, Status: v1.ConditionTrue, Reason: "PodCompleted"}, {Type: v1.PodReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.ContainersReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.PodScheduled, Status: v1.ConditionTrue}, }, ContainerStatuses: []v1.ContainerStatus{ ready(waitingWithLastTerminationUnknown("containerA", 1)), ready(waitingWithLastTerminationUnknown("containerB", 1)), }, Reason:  "Test", Message: "test", }, }, { name: "terminal phase from previous status must remain terminal, restartNever", pod: &v1.Pod{ Spec: v1.PodSpec{ NodeName: "machine", Containers: []v1.Container{ {Name: "containerA"}, {Name: "containerB"}, }, RestartPolicy: v1.RestartPolicyNever, }, Status: v1.PodStatus{ Phase: v1.PodRunning, ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, currentStatus: &kubecontainer.PodStatus{}, previousStatus: v1.PodStatus{ Phase: v1.PodSucceeded, ContainerStatuses: []v1.ContainerStatus{ succeededState("containerA"), succeededState("containerB"), }, // Reason and message should be preserved Reason:  "Test", Message: "test", }, expected: v1.PodStatus{ Phase:    v1.PodSucceeded, HostIP:   "127.0.0.1", QOSClass: v1.PodQOSBestEffort, Conditions: []v1.PodCondition{ {Type: v1.PodInitialized, Status: v1.ConditionTrue, Reason: "PodCompleted"}, {Type: v1.PodReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.ContainersReady, Status: v1.ConditionFalse, Reason: "PodCompleted"}, {Type: v1.PodScheduled, Status: v1.ConditionTrue}, }, ContainerStatuses: []v1.ContainerStatus{ ready(succeededState("containerA")), ready(succeededState("containerB")), }, Reason:  "Test", Message: "test", }, }, { </ins> name: "running can revert to pending", pod: &v1.Pod{ Spec: desiredState,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When a pod is evicted under resource pressure, it goes into an error state. The pod terminates normally. Note that the 130GB file is based on 90% of total disk space - current utilization to hit the default eviction threshold. Full pod spec after going into error state: <details</detailsSee also Eviction tests () are currently broken (), there's a PR up to fix: Kubernetes version (use ): HEAD/1.23.x (kubeletVersion="v1.23.0-alpha.3.47+") Cloud provider or hardware configuration: local-up-cluster OS (e.g: ), Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others: Also reproducible on OpenShift 4.9 (1.23-based) /sig node /cc
mentions this looks very similar to
Confirmed that this is a regression from 1.21. When running the reproducer against the release-1.21 branch, the pod above correctly stays in Evicted status (since pods are mortal). On 1.22, it tries to restart, entering either an Error or ContainerCreating state. /kind regression
/assign
/triage accepted
From : is the first bad commit This was
Seems likely that wasn't actually fixed. The broken eviction tests made it much harder to verify.
Attaching logs from HEAD from reproducer: v=4 Let me also try to get a repro where it goes into Error
Logs from HEAD for reproducer when pod goes into Error: v=4
Roughly, kubelet syncTerminatingPod sends "Failed" to status manager (which looks like it generates the correct apiserver patch for phase running -failed), then somehow by the time Kubelet syncTerminatedPod is called it sends "Running but with errors" to status manager, and status manager allows that transition (internally) and then sends it to the apiserver. So the pod goes Running -Failed -Running which should never happen (we "lost" that the pod was stopped).

Bug 1: pod patch status probably shouldn't allow terminal -nonterminal phase transitions (didn't we fix that??) Bug 2: pod worker is supposed to be getting most up to date status from status manager before calling syncTerminatedPod, which should include "Failed", but that isn't happening? thus the status manager then sends a "non-terminal state" (shouldn't ever happen) Roughly, figure out why status passed to syncTerminatedPod is older than what was sent by syncTerminatingPod, then figure out why status manager allowed that transition back to running, then figure out why apiserver didn't reject.
Staring at logs, I've figured out where the illegal phase transition is happening: If the container failed to properly terminate and generate a status saying as much, generateAPIPodStatus won't have any hints available to it that the pod died because it was killed. (This is happening in the reproducer case because the container won't exit before the termination grace period, and needs to be force killed.) Thus, it will assume that since the container used to be running, its current status is running. This results in an illegal phase transition.
From what I can tell, the issue is that is being passed a pod that still has running containers. From kubelet logs: which gets set as part of . It looks like that will always run so long as succeeds. Thus, I feel like syncTerminatingPod must have the bug: it returns successfully, but the containers of the pod aren't all actually terminated.
/priority critical-urgent based on priority of and the impact of this regression (Kubelet-initiated evictions frequently result in error state pods)
The containers ARE terminated, we just haven't retrieved their status. This is because of a decision to not refresh the container status cache in syncTerminatingPods. So syncTerminatingPod is not a bug.
So then perhaps the right thing to do here would be to refresh the container status cache before syncing another status to the API server in ? Then we can avoid syncing the outdated cached status
Updating from an internal discussion last night - bug 2 isn't actually a bug.

Lantau convinced me we wouldn't refresh the status cache, so what happens is we call generatedAPIStatus with: API status (mostly irrelevant) status manager status to date container cache (but 2s) after The key thing is that phase from runtime status is irrelevant when 1 or 2 are terminal - because runtime has no idea about whether a pod is terminal. So roughly we should be ignoring phase from 3 except when determining when to go from pending to running. The bug here is that we're thinking that this is an "illegal transition" but still copying the latest reason and message (runtime status doesn't get to weigh in on overal terminal phases). We shouldn't need to refresh status at all.
Okay, so there is definitely a bug in what's being synced to the API server for evicted pod statuses right now, possibly for other terminated pods. In 1.21 and earlier, when a pod is evicted, it looks like this: <details</detailsNote that the status field is very minimal, and there are no container statuses. In 1.22+, we attempt to sync container statuses on termination, and sometimes they are wrong (e.g. changing phase from Failed -Running). Here's an example of what an evicted pod could look like on 1.22: <details</details
This appears to be broken even for runOnce pods, tested against latest HEAD (). With spec: After scheduling the pod... And the final pod status: <details</details
Elana identified the issue on status here - previously we NEVER reported container status after an eviction, so the refactor (which correctly ensured container exit codes showed up) was returning containers with state=waiting,lastState=terminated, which is incorrect. generateAPIStatus needs to know the difference between "current container status, waiting for more to show up" and "any status we have is the best we have". So once killPods succeeds in syncTerminatingPods and in syncTerminatedPods, we're no longer "waiting". So generating pod status should correctly summarize the container status as and not inject .


Question: deps = [ "//pkg/controller:go_default_library", "//vendor/github.com/golang/glog:go_default_library", <ins> "//vendor/github.com/juju/ratelimit:go_default_library", </ins> "//vendor/k8s.io/api/certificates/v1beta1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/runtime:go_default_library",

Each node waiting for bootstrapping creates 1 line per 2-3s in the logs when the requesting client doesn't have access. It should not be logging at v(0), these are all v(4) level. We should be reporting these appropriately. I don't want a 5k node cluster showing this.
I'd like to fix it


Question: p.nominatedPods.add(pod, "") p.cond.Broadcast() } <ins> p.receivedMoveRequest = false </ins> return err }

What happened: Kubernetes scheduler may leave a single pod stuck in pending state after a worker node is reloaded/reinstalled. This problem appears to have been introduced in Kubernetes v1.12.4 and v1.13.1. We have not been able to recreate the problem in our testing on earlier releases. The key to the failure is that the worker being reloaded/reinstalled must be the only scheduling option available. The pending pod is eventually scheduled if a new scheduler leader is elected or the node status changes. This appears to be a timing bug since the problem does not always occur. What you expected to happen: No pods are stuck in pending state. How to reproduce it (as minimally and precisely as possible): Start with a single node cluster running a handful of pods. Cordon, drain and then reload/reinstall the node. Since this is a timing bug, it may take several iterations to recreate the problem. Environment: Kubernetes version (use ): 1.12.4, 1.13.1 and 1.13.2 Cloud provider or hardware configuration: VMs running in VirtualBox OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Custom ansible scripts Others: N/A
/sig scheduling
Events from the pending pod: Events from the node:
does "reload/reinstall" mean info is still kept in etcd? or (or equivalent steps) is executed to remove that node entirely? And I assume the "reloaded" node keeps the same spec and IP/node name with previous one?
the info is still kept in etcd. And the node keeps the same spec, IP and name as it had before. The reload does the following: and drain the node the VM the VM again with the same spec the node
What's the taint/toleration accordingly?
there were no taints to the node during this process. The pods are part of a deployment with only basic tolerations provided by Kubernetes. Deleting the pod will get it scheduled even though the taints and tolerations are unchanged. Simply restarting the scheduler leader will get the pod scheduled as well.
Can you share and ?
The symptom (esp. "a single pod stuck in pending state") is quite similar as issue .

I think why this occurs since v1.12.4 is related the PR - which was delivered to master branch, and backported to v1.12.4, and it appears to reveal higher chance of racing. cc/ Basically we have 2 solutions to fix this: the real root cause of the racing condition and give fix (I haven't got luck to reproduce this yet) that we have a solution () to alleviate this kind of problem, could you try to run your test on a) master branch or b) v1.12.4 with cherrypicked to see if the issue is resolved? and does this fits your business need? (in other words, the single pod may be pending for 30 seconds at most)
I'll recreate and provide you the details. However, the pod that is stuck pending is different every time.
Thanks, I'll work on the recreate and test of the cherrypick. That PR certainly seems promising.
Node YAML after recreate:
Pod YAML after recreate:
the cherrypick doesn't look trivial for 1.12 or 1.13. Give me some time to setup an environment to test on 1.14. Unless you have another way?
Yeap, it's not a trivial task to rebuild the source code. Let me know if you need any help. I suppose you're going to try on latest 1.14 tag? e.g. v1.14.0-alpha.1? Right now I'm working on the 1st solution I mentioned in The challenge is I can't reproduce this in my local dev env (I'm running on source code branch of v1.12.4) yet, so my current plan is to retry reproducing your issue to make my solution more convincing. If still no luck, I will send out the PR for reviewing early next week.
And let me confirm with your statement: When the issue occurs, everytime there is only one single pod stuck in pending state, right?
Still don't have luck in reproducing the issue. But I've prepared a PR and put detailed analysis there. Any comments are welcome.
No, the problem does not occur every time. It can take as many as 7 retries before reproducing it.

Sry, I should have made it more clear - I wanted to confirm that among all the times the issue is reproducible, is that everytime there is only one single pod in pending state?
correct, every time that I've reproduced the problem there has been only one pod in pending state.
Thanks. To some extent, it's consistent with the deduction in my PR.
is it possible to have the necessary fixes cherry-picked to at least 1.12 and 1.13? That would make it easier for me to build and test?
Yeap, cherrypick PRs ( onto 1.13, onto 1.12) were created and will be reviewed/approved soon.
Thank you for the cherrypick PRs. I rebuilt 1.13 with the cherrypick PR and still saw the problem on 2 of 10 recreate attempts. Without the fix, I saw a similar failure rate.
the fix is only ready in master right now. Did you rebuild 1.13 by manually running “git cherrypick <this commit id” on release-1.13? Or directly built from latest src code of release-1.13?
I manually rebuilt 1.13.2 with
/reopen I will take another look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've verified is able to fix your issue thoroughly. And cherrypick PRs have been raised (1.12 , 1.13 ).
Thank you.
Thanks, and K8s 13.3 already has another change () that should alleviate the problem significantly. K8s 13.4 will have the definitive fix. Given that the root-cause is fixed, we can close this issue.


Question: proxier.iptablesData.Reset() proxier.iptablesData.Write(proxier.natChains.Bytes()) proxier.iptablesData.Write(proxier.natRules.Bytes()) <ins> proxier.iptablesData.Write(proxier.filterChains.Bytes()) proxier.iptablesData.Write(proxier.filterRules.Bytes()) </ins> glog.V(5).Infof("Restoring iptables rules: %s", proxier.iptablesData.Bytes()) err = proxier.iptables.RestoreAll(proxier.iptablesData.Bytes(), utiliptables.NoFlushTables, utiliptables.RestoreCounters)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After setting up kube-proxy, and static routing. The node is not forwarding the ClusterIP of a service. Services are accessible if the default policy filter FORWARD chain is set to ACCEPT, but not DROP. What you expected to happen: nodes should be forwarding ClusterIP from external cluster to the correctly Pod IP. How to reproduce it (as minimally and precisely as possible): K8S cluser. NGINX and ClusterIP type of service for NGINX. static routing from external. from external with curl. Anything else we need to know?: The filter table function seems dropped since merge of . However the github merge request dosen't seem to relate to removing the functionality. Is this a merge issue? or intentienal? Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
/sig network
Related logging:
/assign
/cc
Hi guys, Altough I'm new to kubernetes, the missing function might be an easy fix. Do you guys mind if I work on it? Jia
PRs are always welcome!
11.6 still has this issue
We didn't cherry-pick this fix to 1.11 branch. As community is going to not maintain 1.11 release, I would suggest you upgrade to 1.12.


Question: func (rc *reconciler) updateStates(volumesNeedUpdate map[v1.UniqueVolumeName]*reconstructedVolume) error { // Get the node status to retrieve volume device path information. <del> rc.updateDevicePath(volumesNeedUpdate) </del> <ins> // Skip reporting devicePath in node objects if kubeClient is nil. // In standalone mode, kubelet is not expected to mount any attachable volume types or secret, configmaps etc. if rc.kubeClient != nil { rc.updateDevicePath(volumesNeedUpdate) } </ins> for _, volume := range volumesNeedUpdate { err := rc.actualStateOfWorld.MarkVolumeAsAttached(

I tried to run a standalone kubelet (without kube-apiserver, so no kubeClient in kubelet), but there is a panic "invalid memory address or nil pointer dereference" The corresponding logs: I expect standalone kubelet to run without panic. Run a standalone kubelet I think this is a problem of passing into without checking if it is nil. Related code snippet: In , passed the nil into : In , passed the nil into : In , used the nil without checking: <detailsI'm only running kubelet, the kubelet version is Kubernetes v1.22.7-rc.0.17+-dirty </details<detailsNot using cloud provider </details<details</details<detailsI installed with source code, the tag is </details<detailscontainerd 1.4.12 </details<details</details
/sig node
/sig storage The key is to know where to put the :)
I would like to work on this
/assign
Just to make sure, is the command to run standalone kubelet just "./kubelet"?
Hi, the full command used to run the standalone kubelet is as below:
/triage accepted /priority important-soon
is passed in and func also should i add a check for nil reference on volumemanager.NewVolumeManager() only?
What should be the default value of , value which should be used when it is .
I'm not familiar with this code, the description looks correct but people familiar with the VolumeManager should be able to say what is the correct behavior when kubelet runs in standalone mode and it doesn't have a client
What should be the default value of kubeClient, value which should be used when it is nil ?
If Kubelet is running in standalone mode, there is no apiserver and should be nil. From my first look, I think the change should be something like above.
We can't just skip updating node objects and still do rest of reconstruction if is nil. This could potentially be unsafe because volumes which are being used may not be reported correctly via node APIs. So we need to rethink - what volume functions are expected to work if api-server is unavailable? Is standalone mode supposed to mount any volumes at all? cc If kubelet is not supposed to be mounting any volumes at all, then shouldn't kubelet not even start volume-manager in standalone mode? Is api-server unavailability permanent or just during startup?

I am not familiar with standalone mode of kubelet and hence the question. /sig node
Could you please give your thoughts on the above comment.
hostPath volumes, certainly
api-server unavailability does not result in a nil kubeClient... a nil kubeClient means the kubelet was started in standalone mode and was told not to talk to a kube-apiserver
if kubeClient is nil, no volume that requires information from an API object is expected to be used (anything using a PVC, configmap, secret, etc)


Question: }) }) <del> It("should have an terminated reason [NodeConformance]", func() { </del> <ins> /* Release : v1.13 Testname: Kubelet, failed pod, terminated reason Description: Create a Pod with terminated state. Pod MUST have only one container. Container MUST be in terminated state and MUST have an terminated reason. */ framework.ConformanceIt("should have an terminated reason [NodeConformance]", func() { </ins> Eventually(func() error { podData, err := podClient.Get(podName, metav1.GetOptions{}) if err != nil {

What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. To my eyes they may meet the criteria for conformance. There's definitely nothing about them that is node specific. They should be running as part of our regular CI, and they should probably be running as presubmits. Porting them to test/e2e/common would cause this to happen. /kind cleanup /sig node /area test /assign
GitHub didn't allow me to assign the following users: mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. /sig node /area test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Expect(logOutput).ToNot(ContainSubstring("stdin closed")) return strings.Contains(logOutput, "abcd1234"), nil }) <del> if err != nil { os.Exit(1) } </del> Expect(err).To(BeNil()) Expect(c.BatchV1().Jobs(ns).Delete("run-test-3", nil)).To(BeNil())

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: ErrCodeResourceVersionConflicts ErrCodeInvalidObj ErrCodeUnreachable <ins> ErrCodeTimeout </ins> ) var errCodeToMessage = map[int]string{

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // in case of latency in our caches, make a call direct to storage to verify that it truly exists or not _, err = e.client.Namespaces().Get(a.GetNamespace()) if err != nil { <del> return admission.NewForbidden(a, fmt.Errorf("Namespace %s does not exist", a.GetNamespace())) </del> <ins> if errors.IsNotFound(err) { return err } return errors.NewInternalError(err) </ins> } return nil

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: if pod.Status.Phase != v1.PodRunning || !podutil.IsPodReady(&pod) { // save this pod name for later, but pretend it doesn't exist for now <del> unreadyPods.Insert(pod.Name) </del> <ins> if pod.Status.Phase != v1.PodFailed { // Failed pods should not be counted as unready pods as they will // not become running anymore. unreadyPods.Insert(pod.Name) } </ins> delete(metrics, pod.Name) continue }

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: // find out what content is supported on the server resources, err := c.Discovery().ServerPreferredNamespacedResources() <del> if err != nil { </del> <ins> if err != nil && !isDynamicDiscoveryError(err) { </ins> return false, err } groupVersionResources, err := discovery.GroupVersionResources(resources) <del> if err != nil { </del> <ins> if err != nil && !isDynamicDiscoveryError(err) { </ins> return false, err }

Seen in https://k8s- Aug 26 09:35:45.074: Couldn't delete ns: "e2e-tests-containers-9cscj": an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get ) (&errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get )", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc4217a4900), Code:500}})
https://k8s-


Question: return wait.ExponentialBackoff(backoff, fn) } <del> func createTestClient() *fake.Clientset { </del> <ins> func createTestClient(attachedVolumes ...v1.AttachedVolume) *fake.Clientset { </ins> fakeClient := &fake.Clientset{} <ins> if len(attachedVolumes) == 0 { attachedVolumes = append(attachedVolumes, v1.AttachedVolume{ Name:       "fake-plugin/fake-device1", DevicePath: "fake/path", }) } </ins> fakeClient.AddReactor("get", "nodes", func(action core.Action) (bool, runtime.Object, error) { return true, &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: string(nodeName)}, Status: v1.NodeStatus{ <del> VolumesAttached: []v1.AttachedVolume{ { Name:       "fake-plugin/fake-device1", DevicePath: "/fake/path", }, }}, </del> <ins> VolumesAttached: attachedVolumes, }, </ins> }, nil <del> }) </del> <ins> }, ) </ins> fakeClient.AddReactor("*", "*", func(action core.Action) (bool, runtime.Object, error) { return true, nil, fmt.Errorf("no reaction implemented for %s", action) })

The following test failures started happening on 30th March: Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (default fs)] volumes should allow exec of files on the volume Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (default fs)] volumes should store data Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (ext4)] volumes should allow exec of files on the volume Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (ext4)] volumes should store data See This seems related to the changes in kubelet fails with the following: Tests passing See the e2e logs above No response <detailsLatest CI build </details<detailsAWS </details<details</details<details</details<details</details<detailsEBS CSI driver 1.5.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig storage
/assign
not having debugging kops builds for awhile, are kubelet logs attached in prow? I don't see them...
Spoke with offline. It looks like we don't have kubelet logs captured in these tests. I need complete kubelet logs to make sense of these errors because testgrid shows inline volume tests as failing and code in question should not be called or inline volumes (because they can not be expanded). If that code fails, it would have been failing before too. I will wait for logs from and in the meanwhile try and see if I can reproduce these failures in single node AWS cluster setup that I can build manually.
thanks
Looks like this is happening for inline volumes if CSI migration is enabled for them. We construct a "fake" object on the fly for migration enabled plugins even though volume is inline, which is causing this condition to fail. It should be simple fix.
opened to fix the problem. The code change is just one line but has tests.


Question: Latest(). Flatten(). TransformRequests(func(req *rest.Request) { <ins> // We need full objects if printing with openapi columns if o.PrintWithOpenAPICols { return } </ins> if o.ServerPrint && o.IsHumanReadablePrinter && !o.Sort { group := metav1beta1.GroupName version := metav1beta1.SchemeGroupVersion.Version

/kind bug /sig cli What happened: kubectl 1.11.0 defaults to (see ), hence, you need to explicitly specify it for each and every single request for a resource managed by an aggregated API server. However, it is even not able to print those resources properly (which was possible before with versions <= 1.10.5): and What you expected to happen: Properly printed and well-formatted output of . (Actually, I do not want to specify but this is what is for). Environment: Kubernetes version (use ):Kernel (e.g. ): MacOS Install tools: Brew
/cc
fixed in
/assign


Question: // TODO check dns record in global dns server }) <del> It("should be able to connect to a federated ingress via its load balancer", func() { </del> <ins> PIt("should be able to connect to a federated ingress via its load balancer", func() { </ins> By(fmt.Sprintf("Waiting for Federated Ingress on %v", jig.ing.Name)) // check the traffic on federation ingress jig.waitForFederatedIngress()

This has surfaced in Ingress and Service e2e tests that manipulate DNS entries. The service controller queries the managed dns entry, thinks that the CNAME doesn't exist and adds a new one. However, clouddns says (through a 409) that the entry already exists. Error message aside, the control flow within the service controller's endpoint_helper prevents a service from being deleted. It's unclear at this point whether there's a race condition. I doubt it given the amount of logging that happens showing that there is no concurrent CNAME entry being created. cc Have you seen this in your refactoring?
Hi I used CoreDNS for most of my testing for service controller refactoring and i did not notice these errors. my local test environment is slightly different than the CI environment. i would do more tests on this issue using CoreDNS and would confirm, if it is an issue also in coredns or just google-clouddns.
If you're not still working on this, please unassign yourself and we'll try to find someone else. Thanks.
This issue was labelled only for sig/multicluster and is thus moved over to kubernetes/federation. If this does not seem to be right, please reopen this and notify us /close


Question: svc *api.Service }{ { <del> name: "spec:v4 ip:v4 dualstack:off", </del> <ins> name: "spec:v4 ip:v4 dualstack:off isIPv6Primary:false", isIPv6Primary:          false, </ins> specExpctPrimary:       true, clusterIPExpectPrimary: true, enableDualStack:        false,

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: <del> #!/usr/bin/env bash # Copyright 2015 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Script to update etcd objects as per the latest API Version. # This just reads all objects and then writes them back as is to ensure that # they are written using the latest API version. # # Steps to use this script to upgrade the cluster to a new version: # https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#upgrading-to-a-different-api-version set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" KUBECTL="${KUBE_OUTPUT_HOSTBIN}/kubectl" # List of resources to be updated. # TODO: Get this list of resources from server once # http://issue.k8s.io/2057 is fixed. declare -a resources=( "endpoints" "events" "limitranges" "namespaces" "nodes" "pods" "persistentvolumes" "persistentvolumeclaims" "replicationcontrollers" "resourcequotas" "secrets" "services" "jobs" "horizontalpodautoscalers" "storageclasses" "roles.rbac.authorization.k8s.io" "rolebindings.rbac.authorization.k8s.io" "clusterroles.rbac.authorization.k8s.io" "clusterrolebindings.rbac.authorization.k8s.io" "networkpolicies.networking.k8s.io" "ingresses.networking.k8s.io" ) # Find all the namespaces. IFS=" " read -r -a namespaces <<< "$("${KUBECTL}" get namespaces -o go-template="{{range.items}}{{.metadata.name}} {{end}}")" if [ -z "${namespaces:-}" ] then echo "Unexpected: No namespace found. Nothing to do." exit 1 fi all_failed=1 for resource in "${resources[@]}" do for namespace in "${namespaces[@]}" do # If get fails, assume it's because the resource hasn't been installed in the apiserver. # TODO hopefully we can remove this once we use dynamic discovery of gettable/updateable # resources. set +e IFS=" " read -r -a instances <<< "$("${KUBECTL}" get "${resource}" --namespace="${namespace}" -o go-template="{{range.items}}{{.metadata.name}} {{end}}")" result=$? set -e if [[ "${all_failed}" -eq 1 && "${result}" -eq 0 ]]; then all_failed=0 fi # Nothing to do if there is no instance of that resource. if [[ -z "${instances:-}" ]] then continue fi for instance in "${instances[@]}" do # Read and then write it back as is. # Update can fail if the object was updated after we fetched the # object, but before we could update it. We, hence, try the update # operation multiple times. But 5 continuous failures indicate some other # problem. success=0 for (( tries=0; tries<5; ++tries )) do filename="/tmp/k8s-${namespace}-${resource}-${instance}.json" ( "${KUBECTL}" get "${resource}" "${instance}" --namespace="${namespace}" -o json > "${filename}" ) || true if [[ ! -s "${filename}" ]] then # This happens when the instance has been deleted. We can hence ignore # this instance. echo "Looks like ${instance} got deleted. Ignoring it" success=1 break fi output=$("${KUBECTL}" replace -f "${filename}" --namespace="${namespace}") || true rm "${filename}" if [ -n "${output:-}" ] then success=1 break fi done if [[ "${success}" -eq 0 ]] then echo "Error: failed to update ${resource}/${instance} in ${namespace} namespace after 5 tries" exit 1 fi done if [[ "${resource}" == "namespaces" ]] || [[ "${resource}" == "nodes" ]] then # These resources are namespace agnostic. No need to update them for every # namespace. break fi done done if [[ "${all_failed}" -eq 1 ]]; then echo "kubectl get failed for all resources" exit 1 fi echo "All objects updated successfully!!" exit 0 </del>

The cluster/update-storage- are not maintained. It will be replaced by We still run the hack/test-update-storage- Its invocation will be removed in We can remove the script. /sig api-machinery /assign
Do you mind if I work on this issue ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
are we ready to pull the plug on these shell scripts?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten