Question: - gocritic text: "ifElseChain: rewrite if-else to switch statement" <ins> # Only packages listed here opt into the strict "exported symbols must be documented". # # Exclude texts from https://github.com/golangci/golangci-lint/blob/ab3c3cd69e602ff53bb4c3e2c188f0caeb80305d/pkg/config/issues.go#L11-L103 - linters: - golint - revive - stylecheck text: comment on exported (method|function|type|const)|should have( a package)? comment|comment should be of the form|comment on exported (method|function|type|const)|should have( a package)? comment|comment should be of the form|exported (.+) should have comment( (or a comment on this block))? or be unexported|package comment should be of the form "(.+)...|comment on exported (.+) should be of the form "(.+)...|should have a package comment path-except: cmd/kubeadm </ins> {{- if not .Hints}} # The following issues were deemed "might be worth fixing, needs to be

Looks like the golangci-lint-pr-hints linter was updated recently to put the feature gate comment ahead of the owner field in All the feature gates in are ordered by the following comments: // owner: // kep: // alpha: // beta: // // Describe the feature gate <declare the feature gateThe new linter changes now requires that the comment describing the feature gate should even precede the owner comment line which goes against the structure followed by the entire file and this needs to be fixed Feature gates should be ordered as follows to maintain parity with the rest of // owner: // kep: // alpha: // beta: // // Describe the feature gate <declare the feature gateUpdate/add new feature gate to No response <details</detailsmaster branch of k8s/k8s <details</detailsN/A <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
!
cc
cc
it was supposed to only apply to cmd/kubeadm.
/sig testing
/assign
Thank you for fixing that.
If I break it, I fix it... :hot_face:


Question: kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' # Command kubectl get pods --sort-by="{metadata.name}" <ins> kubectl get pods --sort-by="{metadata.creationTimestamp}" </ins> ############################ # Kubectl --all-namespaces #

I'd like to be able to sort by a timestamp, e.g. but that fails with an exception: I appreciate that the root of the problem is probably outside of kubernetes, yet I hope you'd agree that sorting by a timestamp is very useful functionality.
Agree with sorting by timestamp.. Just make it clear, is of type: while only these types are allowed to be sorted:


Question: "k8s.io/kubernetes/pkg/kubelet/util/format" ) <del> // ProberResults stores the results of a probe as prometheus metrics. var ProberResults = prometheus.NewGaugeVec( prometheus.GaugeOpts{ </del> <ins> // ProberResults stores the cumulative number of a probe by result as prometheus metrics. var ProberResults = prometheus.NewCounterVec( prometheus.CounterOpts{ </ins> Subsystem: "prober", <del> Name:      "probe_result", Help:      "The result of a liveness or readiness probe for a container.", </del> <ins> Name:      "probe_total", Help:      "Cumulative number of a liveness or readiness probe for a container by result.", </ins> }, []string{"probe_type", <del> "container_name", </del> <ins> "result", </ins> "container", <del> "pod_name", </del> "pod", "namespace", "pod_uid"},

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.


Question: }, } for _, tc := range tests { <del> t.Run(tc.name, func(t *testing.T) { persister := newFakePersister() cfg := map[string]string{} if tc.configToken != nil { cfg = token2Cfg(tc.configToken) } tokenCache := newAzureTokenCache() if tc.cachedToken != nil { tokenCache.setToken(azureTokenKey, tc.cachedToken) } </del> <ins> configModes := []configMode{configModeOmitSPNPrefix, configModeDefault} </ins> <del> fakeSource := fakeTokenSource{ token:        tc.sourceToken, tokenErr:     tc.tokenErr, refreshToken: tc.refreshToken, refreshErr:   tc.refreshErr, } </del> <ins> for _, configMode := range configModes { t.Run(fmt.Sprintf("%s with configMode: %v", tc.name, configMode), func(t *testing.T) { persister := newFakePersister() </ins> <del> tokenSource := newAzureTokenSource(&fakeSource, tokenCache, cfg, configMode, &persister) token, err := tokenSource.Token() </del> <ins> cfg := map[string]string{ cfgConfigMode: string(configMode), } if tc.configToken != nil { cfg = token2Cfg(tc.configToken) } </ins> <del> if fakeSource.tokenCalls != tc.tokenCalls { t.Errorf("expecting tokenCalls: %v, got: %v", tc.tokenCalls, fakeSource.tokenCalls) } </del> <ins> tokenCache := newAzureTokenCache() if tc.cachedToken != nil { tokenCache.setToken(azureTokenKey, tc.cachedToken) } </ins> <del> if fakeSource.refreshCalls != tc.refreshCalls { t.Errorf("expecting refreshCalls: %v, got: %v", tc.refreshCalls, fakeSource.refreshCalls) } </del> <ins> fakeSource := fakeTokenSource{ token:        tc.sourceToken, tokenErr:     tc.tokenErr, refreshToken: tc.refreshToken, refreshErr:   tc.refreshErr, } </ins> <del> if persister.calls != tc.persistCalls { t.Errorf("expecting persister calls: %v, got: %v", tc.persistCalls, persister.calls) } </del> <ins> tokenSource := newAzureTokenSource(&fakeSource, tokenCache, cfg, configMode, &persister) token, err := tokenSource.Token() </ins> <del> if tc.expectErr != "" { if !strings.Contains(err.Error(), tc.expectErr) { t.Errorf("expecting error %v, got %v", tc.expectErr, err) </del> <ins> if token != nil && fakeSource.token != nil && token.apiserverID != fakeSource.token.apiserverID { t.Errorf("expecting apiservierID: %s, got: %s", fakeSource.token.apiserverID, token.apiserverID) </ins> } <del> if token != nil { t.Errorf("token should be nil in err situation, got %v", token) </del> <ins> if fakeSource.tokenCalls != tc.tokenCalls { t.Errorf("expecting tokenCalls: %v, got: %v", tc.tokenCalls, fakeSource.tokenCalls) </ins> } <del> } else { if err != nil { t.Fatalf("error should be nil, got %v", err) </del> <ins> if fakeSource.refreshCalls != tc.refreshCalls { t.Errorf("expecting refreshCalls: %v, got: %v", tc.refreshCalls, fakeSource.refreshCalls) </ins> } <del> if token.token.AccessToken != tc.expectToken.token.AccessToken { t.Errorf("token should have accessToken %v, got %v", token.token.AccessToken, tc.expectToken.token.AccessToken) </del> <ins> if persister.calls != tc.persistCalls { t.Errorf("expecting persister calls: %v, got: %v", tc.persistCalls, persister.calls) </ins> } <del> } }) </del> <ins> if tc.expectErr != "" { if !strings.Contains(err.Error(), tc.expectErr) { t.Errorf("expecting error %v, got %v", tc.expectErr, err) } if token != nil { t.Errorf("token should be nil in err situation, got %v", token) } } else { if err != nil { t.Fatalf("error should be nil, got %v", err) } if token.token.AccessToken != tc.expectToken.token.AccessToken { t.Errorf("token should have accessToken %v, got %v", token.token.AccessToken, tc.expectToken.token.AccessToken) } } }) } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using kubectl, authentication fails with the below error For some reason kubectl is adding a lot of s to the start of the SPN - this can be observed in What you expected to happen: Authentication works as in previous versions of Kubectl How to reproduce it (as minimally and precisely as possible): Create a cluster using Azure AD authentication. Authenticate from a client and wait until the token expires. Use Kubectl and have it automatically refresh the token; it will fail due to the extra 'spn:' entries. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: on-premise OS (e.g: ): macOS 10.14.6 Kernel (e.g. ): Darwin Hostname 18.7.0 Darwin Kernel Version 18.7.0: Mon Feb 10 21:08:45 PST 2020; root:xnu-4903.278.28~1/RELEASEX8664 x86_64 Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
/sig cli
Kubectl version 1.18.0 Kubernetes version 1.16.6
can you confirm whether your original kubeconfig already has in apiserver-id or not? in the mean time, you may try it can convert your kubeconfig from azure module to exec module and do the token thing outside kubectl
At this point if I remove the tokens and try to reauth i get the error described above. I've tried it both ways with apiserver-id having an initially and not, both lead to the same behavior in the end.
i know what's going on now. will prepare a fix. the root cause is that token's target resource (adal.Token.Resource) needs to have prefix and this prefix gets unexpectedly to as well, which gets written back to kubeconfig. So every time kubectl is run, it adds to apiserver-id
/priority important-soon
I'm seeing this with kubectl 1.17.3 and kubernetes 1.16.7.

It's unclear to me what versions are impacted and in what version it is fixed, since only version 1.18 is mentioned.
This still happens on kubectl 1.16.12 and 1.17.8
This still happens on kubectl v1.18.5
same
can you guys share your kubectl version --client and kubeconfig with me? you can find me on kubernetes slack. the slack user name is same as my github handle.
Version 1.17.6 works fine.
Since I was mistaken thinking was still causing this for me, I figured I would just drop a note for others in case it's helpful. Turns out my problem was actually due to and looks to have been resolved as of the v3.3.0 release.
This issue has now reappeared for me - although not necessarily the same root cause. AKS 1.19.0 Authentication works, but it spams frequently.


Question: func (og *operationGenerator) GenerateMountVolumeFunc( waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, <del> actualStateOfWorld ActualStateOfWorldMounterUpdater) (func() error, error) { </del> <ins> actualStateOfWorld ActualStateOfWorldMounterUpdater, isRemount bool) (func() error, error) { </ins> // Get mounter plugin volumePlugin, err := og.volumePluginMgr.FindPluginBySpec(volumeToMount.VolumeSpec)

TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet.

The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks!

Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command.

This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!

Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.

because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days:


Question: } func (ss *scaleSet) ensureVMSSInPool(service *v1.Service, nodes []*v1.Node, backendPoolID string, vmSetName string) error { <ins> klog.V(2).Infof("ensureVMSSInPool: ensuring VMSS with backendPoolID %s", backendPoolID) </ins> vmssNamesMap := make(map[string]bool) // the standard load balancer supports multiple vmss in its backend while the basic sku doesn't

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using Internal Standard LoadBalancer on Azure, although VMSS instances have been to LB backend address pools, the VMSS itself is not . Hence, if customer upgrades these VMSS instances to latest model, the backend address pool would be lost and hence the LoadBalancer services would be broken. What you expected to happen: Azure VMSS is also to internal SLB backend address pool automatically. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /kind bug /priority critical-urgent /area provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /priority critical-urgent /area provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
Can you share some insights when this could be get fixed?


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package services import ( "io/ioutil" "os" "os/signal" "syscall" "github.com/golang/glog" ) // e2eService manages e2e services in current process. type e2eServices struct { rmDirs []string // statically linked e2e services etcdServer   *EtcdServer apiServer    *APIServer nsController *NamespaceController } func newE2EServices() *e2eServices { return &e2eServices{} } // terminationSignals are signals that cause the program to exit in the // supported platforms (linux, darwin, windows). var terminationSignals = []os.Signal{syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT} // run starts all e2e services and wait for the termination signal. Once receives the // termination signal, it will stop the e2e services gracefully. func (es *e2eServices) run() error { defer es.stop() if err := es.start(); err != nil { return err } // Wait until receiving a termination signal. sig := make(chan os.Signal, 1) signal.Notify(sig, terminationSignals...) <-sig return nil } // start starts the tests embedded services or returns an error. func (es *e2eServices) start() error { glog.Info("Starting e2e services...") err := es.startEtcd() if err != nil { return err } err = es.startApiServer() if err != nil { return err } err = es.startNamespaceController() if err != nil { return nil } glog.Info("E2E services started.") return nil } // stop stops the embedded e2e services. func (es *e2eServices) stop() { glog.Info("Stopping e2e services...") // TODO(random-liu): Use a loop to stop all services after introducing // service interface. glog.Info("Stopping namespace controller") if es.nsController != nil { if err := es.nsController.Stop(); err != nil { glog.Errorf("Failed to stop %q: %v", es.nsController.Name(), err) } } glog.Info("Stopping API server") if es.apiServer != nil { if err := es.apiServer.Stop(); err != nil { glog.Errorf("Failed to stop %q: %v", es.apiServer.Name(), err) } } glog.Info("Stopping etcd") if es.etcdServer != nil { if err := es.etcdServer.Stop(); err != nil { glog.Errorf("Failed to stop %q: %v", es.etcdServer.Name(), err) } } for _, d := range es.rmDirs { glog.Info("Deleting directory %v", d) err := os.RemoveAll(d) if err != nil { glog.Errorf("Failed to delete directory %s.n%v", d, err) } } glog.Info("E2E services stopped.") } // startEtcd starts the embedded etcd instance or returns an error. func (es *e2eServices) startEtcd() error { glog.Info("Starting etcd") dataDir, err := ioutil.TempDir("", "node-e2e") if err != nil { return err } // Mark the dataDir as directories to remove. es.rmDirs = append(es.rmDirs, dataDir) es.etcdServer = NewEtcd(dataDir) return es.etcdServer.Start() } // startApiServer starts the embedded API server or returns an error. func (es *e2eServices) startApiServer() error { glog.Info("Starting API server") es.apiServer = NewAPIServer() return es.apiServer.Start() } // startNamespaceController starts the embedded namespace controller or returns an error. func (es *e2eServices) startNamespaceController() error { glog.Info("Starting namespace controller") es.nsController = NewNamespaceController() return es.nsController.Start() } // getServicesHealthCheckURLs returns the health check urls for the internal services. func getServicesHealthCheckURLs() []string { return []string{ getEtcdHealthCheckURL(), getAPIServerHealthCheckURL(), } } </ins>

In node e2e, we have the option to keep the services running after test for further debug. However, we set to for the services, which means that once the parent exits, they will receive and exit with the parent if they don't trap the signal. With this, option will never work. If we still need this feature, we should not set if is false. /cc
We still need this feature. It helps with debugging test failures by keeping all the daemons running upon failure. , Lantao Liu wrote:


Question: epmetrics.DroppedRequests.WithContext(ctx).WithLabelValues(epmetrics.ReadOnlyKind).Inc() } epmetrics.RecordRequestTermination(r, requestInfo, epmetrics.APIServerComponent, http.StatusTooManyRequests) <del> if isWatchRequest { close(resultCh) } </del> tooManyRequests(r, w) } <del> // For watch requests, from the APF point of view the request is already // finished at this point. However, that doesn't mean it is already finished // from the non-APF point of view. So we need to wait here until the request is: // 1) finished being processed or // 2) rejected if isWatchRequest { if forgetWatch != nil { forgetWatch() } err := <-resultCh if err != nil { panic(err) } } </del> }) }

As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI


Question: It("should not be able to pull image from invalid registry [NodeConformance]", func() { image := "invalid.com/invalid/alpine:3.1" <del> imagePullTest(image, false, v1.PodPending, true) </del> <ins> imagePullTest(image, false, v1.PodPending, true, false) </ins> }) It("should not be able to pull non-existing image from gcr.io [NodeConformance]", func() { image := "k8s.gcr.io/invalid-image:invalid-tag" <del> imagePullTest(image, false, v1.PodPending, true) </del> <ins> imagePullTest(image, false, v1.PodPending, true, false) </ins> }) <del> // TODO(claudiub): Add a Windows equivalent test. </del> It("should be able to pull image from gcr.io [LinuxOnly] [NodeConformance]", func() { image := "gcr.io/google-containers/debian-base:0.4.1" <del> imagePullTest(image, false, v1.PodRunning, false) </del> <ins> imagePullTest(image, false, v1.PodRunning, false, false) }) It("should be able to pull image from gcr.io [NodeConformance]", func() { framework.SkipUnlessNodeOSDistroIs("windows") image := "gcr.io/kubernetes-e2e-test-images/windows-nanoserver:v1" imagePullTest(image, false, v1.PodRunning, false, true) </ins> }) It("should be able to pull image from docker hub [LinuxOnly] [NodeConformance]", func() { image := "alpine:3.7" <del> imagePullTest(image, false, v1.PodRunning, false) </del> <ins> imagePullTest(image, false, v1.PodRunning, false, false) </ins> }) <del> It("should be able to pull image from docker hub [WindowsOnly] [NodeConformance]", func() { </del> <ins> It("should be able to pull image from docker hub [NodeConformance]", func() { </ins> framework.SkipUnlessNodeOSDistroIs("windows") <ins> // TODO(claudiub): Switch to nanoserver image manifest list. </ins> image := "e2eteam/busybox:1.29" <del> imagePullTest(image, false, v1.PodRunning, false) </del> <ins> imagePullTest(image, false, v1.PodRunning, false, true) </ins> }) It("should not be able to pull from private registry without secret [NodeConformance]", func() { image := "gcr.io/authenticated-image-pulling/alpine:3.7" <del> imagePullTest(image, false, v1.PodPending, true) </del> <ins> imagePullTest(image, false, v1.PodPending, true, false) </ins> }) It("should be able to pull from private registry with secret [LinuxOnly] [NodeConformance]", func() { image := "gcr.io/authenticated-image-pulling/alpine:3.7" <del> imagePullTest(image, true, v1.PodRunning, false) </del> <ins> imagePullTest(image, true, v1.PodRunning, false, false) }) It("should be able to pull from private registry with secret [NodeConformance]", func() { framework.SkipUnlessNodeOSDistroIs("windows") image := "gcr.io/authenticated-image-pulling/windows-nanoserver:v1" imagePullTest(image, true, v1.PodRunning, false, true) </ins> }) }) })

There is now a test tag from I remember a tag being approved in SIG Architecture and SIG Testing, but I don't remember a tag. If this was approved I think we need to broadcast it more widely as many existing test setups including current conformance testing appear to now need to skip / exclude this tag. cc /area conformance /sig architecture /sig testing
/assign
bug may not be the appropriate type, but seemed like the closest issue template to start from initially.
TL;DR nope, that was not something that I'm aware of.
cc
The email thread is in
the in that thread says: That's not the case here.
we should file a PR marked 1.14 and to yank that tag, just so we don't forget. Of course if the decision changes in the email thread, we can revisit the PR. ok?
makes sense to me
Catching up. Quick comment before I read the threads: Documented kinds of tests are currently here: WindowsOnly had been hypothesized, but not yet to that list.
Also, are there already tests tagged as SIG Windows?
On conformance tests: Windows isn't part of conformance yet, so no conformance test should have this tag. And no conformance test should contain any Skip directives. They should die in that case:
I looked at and don't understand why the Windows image test wasn't written as a separate test. To avoid either refactoring or copy/pasting? It looks like a pretty obscure approach -- auto-skipping individual test cases that contain a particular tag?
Pretty much. But it is no longer the case, since all those "should pull image from X" test cases have been refactored. (look at to see how it looks now)
Yes, that's better, thanks. To answer my own question regarding , here's an example: which also invokes inside the test. So we're agreed that we don't need yet?
now removes all instances of the [WindowsOnly] tag.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : /sig testing /sig windows In the the PR we're adding a test case for pulling a Windows image from dockerhub (. However, there are no known public Windows images on , so the only option would be to upload it ourselves. Our proposal is to have it uploaded to the using Windows images). The proposed image manifest list to be uploaded is built from [1], which is currently hosted on dockerhub [2] and is currently used for testing Kubernetes with Windows nodes [3]. The manifest list contains the images: e2eteam/busybox:1.29-1803 (for Windows Server 1803) e2eteam/busybox:1.29-1809 (for Windows Server 2019) [1] [2] [3] Why is this needed: The Windows images will be used for the Windows equivalent test case for the following test:
/assign
/area conformance
It is my contention that we really only want one prod repo and that should be folded into the main repo. The authenticated repo should be for testing authenticated pulls and nothing else. To that end, I propose that we start with this. Does the image need to be named or tagged or something to indicate OS? How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Or can we have a multi-os manifest list? Also, is busybox the right image for testing, or can we do something even more minimal (to address the "what f there's a busybox CVE" question)?
Yes, (all images to date have used ), so I'd prefer that route over yet another name or tag. I'd also prefer something other than busybox if possible. I originally created because there were so many images in and it was hard to tell which of them were production images, which were test images, and which had been completely abandoned. Also, only a small number of Googlers could push to this repo since it contained all the production images. To address the first issue, I decided that all e2e test images would be sourced from the directory. To address the second issue, I created a new project with a wider ACL (though still currently limited to Googlers).

All that said, given the automation we're building for the main GCR repo, it'd probably be fine to fold these images back in.
The image promoter means that a single repo should be BETTER managed than individual repos - we will have a PR trail of who put images there and why.
We should probably broaden the discussion to non-busybox Windows test images too. (related: ). Do we have a lower bar for test images? If so, having a separate repo (or making it clear that the images are test only) does have clear benefits.
Do we want a lower bar? I don't think we do.
Nobody wants a lower bar until they want to add/update the image quickly for a new test, or fix a test failure to unblock the release. If we do effectively hold the same bar today, this shouldn't be a problem.
i agree. also we don't need schema v2. multi-os, multi-arch image here. just (e.g.) a nanoserver image that satisfies a Windows 1803 and Windows 1809 host. a nanoserver image will obviously imply Windows only. for a test called "should be able to pull image from [WindowsOnly]": all we need is to upload this image to [x] and i guess we need to decide on if we want this test to be / passing for Windows Containers GA.
And indeed, a manifest list with nanoserver images could be an alternative (considering that the ) (we did try to find a command which works on both OSes, and ultimately found curl, but not all Linux images contain curl, check PR ).
an official nanoserver image is a more secure download. do we need to build a nanoserver manifest list or are there official images that we can use? i would like to bring a topic here related to pause. why are all the pause images for Windows that i found 1.1 GB or more? we definitely need the absolute minimal pause container uploaded in a location where users can download it and use that location in the guide. AFAIK, 100 MB is as low as we can go for a native Windows container, but that's still much better... i brought this topic in the other PR. we don't need a command that works on both OSes.

if is the command to use on Windows containers only, so be it.
And also, my image? It might be out-of-date.
we need to start planning how Windows containers can used be easily on Windows nodes. the pause image should be pre-built, small and easy to pull by deployment tools, such as kops, kubeadm. that is much better, but still a bit too much? can we go down to ~100MB? this image size was reported by a user in a k/k issue recently: my assumption is that the user was building the image himself.
i am ok to have that conversation, but please file a ticket and let's discuss that in sig-windows
everyone, thank you for your help and responses. trying to bring back this conversation to the original request. what would you like us to do in order for the few folks from google that have permissions to upload the windows images to be used for testing.
I am happy to help publish this via the alias, which I believe is the correct long-term path. We need an image - preferably not one as prone to CVEs as busybox historically is. Hence a suggestion for "pause" or something equivalently useless and tiny (if it is tiny). Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Michael Michael <:
So, I'll sumarize the previous discussions. The proposed images are: The image currently named image tags:. based on: respectively. image is currently used in Windows E2E tests and should already exist on the nodes, so download should be minimal. The image. a manifest list needs to be created containing the tags: The image is used to tag the image, which is used by kubelet, so it should already exist on the nodes. smaller in size (~350MB).

the test should create pods with the command . I'm fine with either one, although I think people will tend towards the nanoserver image, and in fairness, I think it will be the easiest one to maintain long-term. AFAIK, Microsoft updates their container images periodically, so, we should do the same for the image. If not, the test using the image will then have to pull the entire image (350 MB is ok, IMO). E.g. vs Well, a manifest list can contain both Linux and Windows images, So, if the busybox image is picked, the Linux busybox image can be included in the manifest as well. Ideally, most images work on both OSes (with some exceptions, like the nanoserver or alpine images), so I don't think there's a need to put windows in the path. And if a new root path is being for new test images, I would like to ask that images mustn't container slashes in their names (e.g.:). Docker doesn't support images with slashes in their names (see ). Even if all the images on eventually support Windows, it would still be a good idea to be able to use dockerhub for testing purposes.
We need an image - preferably not one as prone to CVEs as busybox WHOAH. That's a 2GB image. For testing. I am in favor of this just based on the size (I was thinking something like kuberneets' pause, which is 240 kB. Over 1000 times smaller. We have different comprehensions of what "nano" means. 350 MB is not OK. It's acceptable if we have no other choice, but honestly we should create our own that is as small as humanly possible. Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? I don't have enough context on e2e future, so I'd like some input from someone who does. Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g.

vs I agree (except for busybox, which is a known vector for contaigion). Having a multi-os, multi-arch "serve_hostname" or something equally dumb seems ideal. Docker HUB doesn't. Docker itself does, and GCR does. If you're using dockerhub for testing, you have to change "" to "e2eteam/whatever", right? So what does it matter if the GCR path is longer? That said, I would be OK to have a single dir if that is what test-team wants. It just seems likely to get messy eventually.
Yes, dockerhub, I forgot the "hub" :) And yes, that's what we're doing and it works for the most part. But, if you take a look here [1], basically, the root is , which, as you know, is not accepted by dockerhub. So, the proposal was to not have "image names" with slashes in them (e.g.: in this case). [1]
I see. I think it would be OK to ask that images don't use subdirs, but the e2eRegistry variable could. e.g. // Docker hub e2eRegistry = "e2eteam" // GCR e2eRegistry = "" ResourceController = Config{e2eRegistry, "resource-consumer-controller", "1.0"} That seems workable, though it doesn't answer the question of the actual path in GCR :) Claudiu Belu <:
That seems workable, though it doesn't answer the question of the Yep, exactly right. :)
what are the next steps on this ticket to get the images to
I am still waiting for which image we are going to use, and for any objections to "" as a root. Michael Michael <:
Well, I think we are in agreement that we should use the nanoserver images. As for the root repo, I have no objections.
can you please explicitly give all the images to upload in an authoritative list? i am hoping we can upload all of these by monday EOD.
Sure. As we've discussed previously, the images are:
no manifest list image that supports both 1809 and 1803? are we OK for only having 1809 for that "pull from GCR" test?

Yeah, I meant those images should be included in a single manifest list (something like or something), like we've previously discussed. As for the second question, AFAIK, the GCE Windows CI is using Windows Server 1803 at the moment, so it would be nice to have the new test running on that too.
see above for the two images we want to push. we also need to repush this image resource-consumer/controller:1.0 to avoid the "/" as resource-consumer-controller cc:
i pulled these images on Windows: size of each is roughly ~350MB. and then saved them to TAR and uploaded them here: to go lower than that we might have hack custom images that trim, for instance a bunch of DLLs that we don't need (for e.g. a minimal pause image), however such a custom image will just require yet another unique download if the users are already basing their workload containers on official images. also for such hacks we need someone to verify if the nanoserver LICENSE allows that. i'm pretty sure it doesn't.
no hacks please. we need to upload the images as they officially come from microsoft. :)
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
The slash thing is kinda irrelevant to me, it's just a limitation of Docker Hub as pointed out
i understand that. but can you make forward progress in testing that autoscale ticket without this work?
i think you have the two images to upload. let us know when this is complete and thank you so much!
The tars shouldn't be necessary. The nanoserver images include 2 layers that are type and should not be pushed. here's an example manifest from the bazel repo. you could probably use the existing rulesdocker to push nanoserver with no changes if you're building and pushing images with bazel
we can't follow this process due to Windows licensing. i will hide your post so that it is not followed by accident.
just to clarify - you can reference those in manifest lists, and docker push / pull them without actually pushing the layer contents to a manifest v2 complaint registry. That's the right way to do it.

That separates out user-owned files in the upper layers from those in the base layers that are always served from a Microsoft server due to licensing restrictions
Oh ok then. Keeping in mind the fact that the nanoserver image is a manifest itself (for amd64 and arm) and that you cannot reference another manifest list in your manifest list, then the the actual images which should be to the manifest list are: 1803 doesn't have multiarch.
the post is still quoted in your response. best to edit that as well? ok, looks like the licensing of these image is slightly more complicated than expected... i hope a tutorial is not required when pushing to GCR. :)
What is the status of this now? I am THOROUGHLY confused
me too according to here and here and as far as i understood, we only have to push the JSON files that hold layer references, but not the actual layers because this violates the licensing of the nanoserver images.
I'll post explicit steps using , etc.
a PowerShell window as an administrator Docker-CE 18.09.2 experimental mode for the Docker CLI Here's the copy & paste steps to pull the existing images, push them to the new registry, and create a manifest. Change $destinationRepo to be something like or your private repo name. The pushes will be nearly instantaneous because they're manifest-only pushes. All layer data is skipped. Now, you can inspect the manifest and it will show multiple images with different os.version And last, push it
Here's the example repo I made on When things are done this way, the original Windows image is still served from Microsoft as required by the license. If I were to add unique layers on top of this image - that content would be pushed to the registry.
Pushed to my staging, will try to get the promoter running ASAP
hmm, it looks like our image promoter is not quite ready for multi-platform yet. Let me try pushing to
yeah, you might be bumping into
Someone test that pull?
does that mean no Windows images can live on GCR? Am I misreading it?
I believe GCR supports it, but some of our tooling (which uses google/containerregistry) may not. OTOH I think tool is just using directly, so it might be unaffected.
!
Awesome, so that seems to work for now.

I guess the next step is to FROM that in your own containers.
or just use it in your tests. Do we close this for now, then?
SGTM, leaving the close call to
This looks right. I did a pull on a 1809 machine and the hashes match. is this enough to finish that test case, or are more images needed? We definitely need to get the images into a CI for 1.15. See /
this should be enough in terms of the test case.
^ FYI, there are now Windows test images at GCR. thanks !
Awesomeness, I've made a PR adding the testcase using this new image. This issue will close automatically once that merges. Test works for me. One question remains though. Do we have to have a Windows equivalent for the , so it fails on Windows. If we have to add a Windows equivalent for that too, then the nanoserver manifest list should also be pushed to the registry as well. Anyways. thanks for the help, everyone!
are you asking if can push the nanoserver manifest to the authenticated-image-pulling registry as well? Tim?
Yep. It is a conformance test, so it would be nice to be able to cover it as well.
any chance you push the nanoserver manifest to the authenticated-image-pulling registry as well? thanks for all your help! once you do the above, we can lose this issue
Try ""
thanks it works! !
please, close the issue once you check this.
thanks Tim! you rock!
can you please close this issue if we are done?
The issue is supposed to close when the PR merges. Closing anyways.


Question: } } <del> // StatefulSetHasDesiredReplicas returns a conditon that checks the number of statefulset replicas </del> <ins> // StatefulSetHasDesiredReplicas returns a condition that checks the number of StatefulSet replicas </ins> func StatefulSetHasDesiredReplicas(ssClient appsclient.StatefulSetsGetter, ss *apps.StatefulSet) wait.ConditionFunc { <del> // TODO: Differentiate between 0 statefulset pods and a really quick scale down using generation. </del> <ins> // If we're given a StatefulSet where the status lags the spec, it either means that the // StatefulSet is stale, or that the StatefulSet manager hasn't noticed the update yet. // Polling status.Replicas is not safe in the latter case. desiredGeneration := ss.Generation </ins> return func() (bool, error) { ss, err := ssClient.StatefulSets(ss.Namespace).Get(ss.Name, metav1.GetOptions{}) if err != nil { return false, err } <del> return ss.Status.Replicas == ss.Spec.Replicas, nil </del> <ins> // There's a chance a concurrent update modifies the Spec.Replicas causing this check to // pass, or, after this check has passed, a modification causes the StatefulSet manager to // create more pods. This will not be an issue once we've implemented graceful delete for // StatefulSet, but till then concurrent stop operations on the same StatefulSet might have // unintended side effects. return ss.Status.ObservedGeneration != nil && *ss.Status.ObservedGeneration >= desiredGeneration && ss.Status.Replicas == ss.Spec.Replicas, nil </ins> } }

StatefulSet now as of 1.6. That means the TODO on can be fixed in 1.7. We could not fix it in 1.6 because that code is used by kubectl, which must work against servers that are one minor version ahead of or behind the client.
/cc
I am volunteered to take this job if no one start working on this.
I don't think anyone's working on it, so you are welcome to take it. Feel free to tag me as a reviewer. Thanks!


Question: return false, err } _, err = vs.InstanceID(ctx, convertToK8sType(nodeName)) <del> if err == nil { return true, nil </del> <ins> if err != nil { if err == cloudprovider.InstanceNotFound { return false, nil } return false, err </ins> } <del> return false, err </del> <ins> return true, nil </ins> } // InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have k8s cluster with version . And I noticed that if I deleted worker node VM from IAAS, then node stuck in state, I created statefulset before and the pod scheduled on that node was stuck in state, and no new pod was created to replace that. Node should be deleted if no instance is found Related log When I check the log of I could see following log keeps popping up. Looks like node_controller noticed that the VM is gone but it doesn't delete the node. Related code here will return if no instance is found. But it should check if it's VM is deleted not shutdown, then return And it should cherry pick this fix avoiding deleting shutdown node What you expected to happen: After VM is deleted, the node should be deleted, pods assigned to the node should be rescheduled to other available nodes How to reproduce it (as minimally and precisely as possible): k8s cluster. In our case we use PKS. workloads on node worker node VM from IAAS Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere Cloud Provider
/sig cloud-provider /sig vmware
Same issue as this one The issue and is cluster API repo. We should fix it inside node_controller in k/k
Hi this issue was found when testing windows cluster deployed using PKS. We do have a workaround by manually deleting node using kubectl. Still we wish the fix could go in ASAP. Could you ask someone from upstream to to review the PR that fix this issue? Thank you
, we will try to push this through for v1.20. 1.19 is approaching code thaw, so we can't push this through
Thanks Michael. I'll let my team know.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: proxy := httputil.NewSingleHostReverseProxy(&url.URL{Scheme: h.Location.Scheme, Host: h.Location.Host}) proxy.Transport = h.Transport proxy.FlushInterval = h.FlushInterval <del> proxy.ServeHTTP(maybeWrapFlushHeadersWriter(w), newReq) } // maybeWrapFlushHeadersWriter wraps the given writer to force flushing headers prior to writing the response body. // if the given writer does not support http.Flusher, http.Hijacker, and http.CloseNotifier, the original writer is returned. // TODO(liggitt): drop this once https://github.com/golang/go/issues/31125 is fixed func maybeWrapFlushHeadersWriter(w http.ResponseWriter) http.ResponseWriter { flusher, isFlusher := w.(http.Flusher) hijacker, isHijacker := w.(http.Hijacker) closeNotifier, isCloseNotifier := w.(http.CloseNotifier) // flusher, hijacker, and closeNotifier are all used by the ReverseProxy implementation. // if the given writer can't support all three, return the original writer. if !isFlusher || !isHijacker || !isCloseNotifier { return w } return &flushHeadersWriter{w, flusher, hijacker, closeNotifier} } type flushHeadersWriter struct { http.ResponseWriter http.Flusher http.Hijacker http.CloseNotifier } func (w *flushHeadersWriter) WriteHeader(code int) { w.ResponseWriter.WriteHeader(code) w.Flusher.Flush() </del> <ins> proxy.ServeHTTP(w, newReq) </ins> } // tryUpgrade returns true if the request was handled.

The workaround in should be removed (but the test kept to prevent regressions) once we upgrade to go 1.12.2 and pick up /assign
Thanks Liggitt
/sig api-machinery
removed in


Question: t.Fatalf("expected one pod to be unmanaged pod but found %d", len(unmanagedPods)) } ps.VerifyNoStatusError(t, pdbName) <ins> verifyUnsupportedControllerEventEmitted(t, dc) </ins> } // Verify that we count the scale of a ReplicaSet even when it has no Deployment.

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Write creates or overwrites the contents of the version.txt file with the given EtcdVersionPair. func (v *VersionFile) Write(vp *EtcdVersionPair) error { <del> data := []byte(fmt.Sprintf("%s/%s", vp.version, vp.storageVersion)) return ioutil.WriteFile(v.path, data, 0666) </del> <ins> // We do write + rename instead of just write to protect from version.txt // corruption under full disk condition. // See https://github.com/kubernetes/kubernetes/issues/98989. err := ioutil.WriteFile(v.nextPath(), []byte(vp.String()), 0666) if err != nil { return fmt.Errorf("failed to write new version file %s: %v", v.nextPath(), err) } return os.Rename(v.nextPath(), v.path) </ins> } func exists(path string) (bool, error) {

file was found 'empty'. It blocked running etcd in a fail loop with Before this issue: Data disk (etcd) run out of space due to . .txt be not corrupted. It should be updated: only when needed (infrequently) safely (rename swap or writing on pre-allocated space) migrator script that runs etcd in case of the file not being found. Use whole space of etcd data disk and restart migrator+etcd when concurrent process is disk hungry. Requires changes of this logic: This logic (file write) should not trigger if the minor version matches: The write itself: should be defensive: Please notice that truncates file before writingKubernetes version: 1.18 Cloud provider or hardware configuration: GCE
/sig api-machinery
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen


Question: numVolumes++ allErrs = append(allErrs, validateAzureFile(pv.Spec.AzureFile, specPath.Child("azureFile"))...) } <ins> if pv.Spec.VsphereVolume != nil { if numVolumes > 0 { allErrs = append(allErrs, field.Forbidden(specPath.Child("vsphereVolume"), "may not specify more than 1 volume type")) } else { numVolumes++ allErrs = append(allErrs, validateVsphereVolumeSource(pv.Spec.VsphereVolume, specPath.Child("vsphereVolume"))...) } } </ins> if numVolumes == 0 { allErrs = append(allErrs, field.Required(specPath, "must specify a volume type")) }

vSphere volume spec fails with the following error, as its not been in api/validation. "spec.volumes[0]: Required value: must specify a volume type" Fixed in PR


Question: "errors" "net" "reflect" <ins> "strings" </ins> "sync" "time"

NodeController is unable to register the value of , because it contains mixed case from the OS (in this case OSX). I'm not positive that all Linux systems are guaranteed to lowercase hostname -f, but it's likely that automated processes may end up collecting hostnames from sources that preserve case and passing that to the kubelet or master. I believe we should lowercase names detected from hostname -f and the values the node controller users - it's a minor thing, but internally we can guarantee that we treat node names as insensitive and in their canonical interior form.
cc
I'll take a look tmr.


Question: cmd.Flags().Set("output", defaultOutputFormat) options.Merge.Default(true) <del> cmd.Flags().Var(&options.Merge, "merge", "merge together the full hierarchy of kubeconfig files") </del> <ins> f := cmd.Flags().VarPF(&options.Merge, "merge", "", "merge together the full hierarchy of kubeconfig files") f.NoOptDefVal = "true" </ins> cmd.Flags().BoolVar(&options.RawByteData, "raw", false, "display raw byte data") cmd.Flags().BoolVar(&options.Flatten, "flatten", false, "flatten the resulting kubeconfig file into self contained output (useful for creating portable kubeconfig files)") cmd.Flags().BoolVar(&options.Minify, "minify", false, "remove all information not used by current-context from the output")

It acts like a boolean flag but requires an argument, unlike boolean flag is pretty much every environment. We need to find a better way to express the things that this is expressing. Maybe a add a tristate to the pflag lib that behaves the same way but doesn't demand an argument?
cc
The handles "true", "false", "unset" and has a default value. I seem to recall you were looking at changing it, but we needed to preserve the value. The complain here is that you have to do instead of just . /cc
fixing is really easy....


Question: buildBackOffDuration = time.Minute syncLoopFrequency    = 10 * time.Second maxBackOffTolerance  = time.Duration(1.3 * float64(kubelet.MaxContainerBackOff)) <ins> podRetryPeriod       = 1 * time.Second podRetryTimeout      = 1 * time.Minute </ins> ) // testHostIP tests that a pod gets a host IP

[x] Original approval issue: 09 July 2020 [x] Two weeks soak end date: 24 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Pods with a static label within a Namespace. that all 3 Pods with the label in the Namespace are created. the set of Namespaced Pods with a label via DeleteCollection. that all Pods with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Test to be created using "Polling" and not "Watch tooling"
/reopen Merged Issue open, waiting for Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged Issue open, waiting for Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The test appears to be flaking, need to reach out to #sig-node for some feedback.
Can you point us to the link of the flaky tests? Thanks!
The Promotion The testgrid Thank you!


Question: EXTRA_DOCKER_OPTS="${EXTRA_DOCKER_OPTS:-}" VOLUME_PLUGIN_DIR="${VOLUME_PLUGIN_DIR:-/home/kubernetes/flexvolume}" <del> REMOUNT_VOLUME_PLUGIN_DIR="${REMOUNT_VOLUME_PLUGIN_DIR:-true}" </del> SERVICE_CLUSTER_IP_RANGE="${SERVICE_CLUSTER_IP_RANGE:-10.0.0.0/16}"  # formerly PORTAL_NET ALLOCATE_NODE_CIDRS=true

/kind bug https://k8s- This is likely caused by a missing REMOUNTVOLUMEPLUGIN_DIR flag in kube-env construction. /assign /sig storage
xref /priority failing-test /priority critical-urgent any update on this one?
/status approved-for-milestone
I'm aware of the solution, but I'm trying to understand how to limit the scope of the change to just COS images, instead of all VM distros.
xref
Fix submitted, awaiting review:
PR approved; awaiting 1.10 milestone marker
/status in-progress
You must be a member of the github team to add status labels.
/status in-progress
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action Required: This issue has not been updated since Mar 11. Please provide an update. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } if creating { err = storeerr.InterpretCreateError(err, qualifiedResource, name) <del> err = rest.CheckGeneratedNameError(e.CreateStrategy, err, creatingObj) </del> <ins> err = rest.CheckGeneratedNameError(ctx, e.CreateStrategy, err, creatingObj) </ins> } else { err = storeerr.InterpretUpdateError(err, qualifiedResource, name) }

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: nodesToUpdate := nsu.actualStateOfWorld.GetVolumesToReportAttached() for nodeName, attachedVolumes := range nodesToUpdate { nodeObj, err := nsu.nodeLister.Get(string(nodeName)) <del> if nodeObj == nil || err != nil { // If node does not exist, its status cannot be updated, log error and // reset flag statusUpdateNeeded back to true to indicate this node status // needs to be updated again </del> <ins> if errors.IsNotFound(err) { // If node does not exist, its status cannot be updated. // Remove the node entry from the collection of attach updates, preventing the // status updater from unnecessarily updating the node. </ins> glog.V(2).Infof( <del> "Could not update node status. Failed to find node %q in NodeInformer cache. %v", </del> <ins> "Could not update node status. Failed to find node %q in NodeInformer cache. Error: '%v'", </ins> nodeName, err) <ins> nsu.actualStateOfWorld.RemoveNodeFromAttachUpdates(nodeName) continue } else if err != nil { // For all other errors, log error and reset flag statusUpdateNeeded // back to true to indicate this node status needs to be updated again. glog.V(2).Infof("Error retrieving nodes from node lister. Error: %v", err) </ins> nsu.actualStateOfWorld.SetNodeStatusUpdateNeeded(nodeName) continue }

kubernetes version: 1.5.3 platform: aws deploy tool: kops my master's /var/log/kube-controller- is receiving this type of log: lots of times per second, this node "ip-172-20-114-85.ec2.internal" does not exist in the cluster anymore and is not shown if I shouldn't it be removed from status_updater?
Any work around? Any way to purge those old nodes from the NodeInformer cache?
AFAIK, sig-node doesn't maintain the node controller. Based on the files, the only approver is
It has nothing to do with the NodeController: cc
This was happening in our clusters as well, only the leader was doing it and and killing the container so it restarted made it stop happening.
did you kill the container, which then gets restarted by the scheduler, and that fixed it?
correct
Works for me
Same controller-manager log here using k8s 1.5. I deleted one of the nodes which ran out of resources and became stuck, something I'm still trying to pinpoint the cause of but it appears to be related to volumes.
This happens on k8s 1.6.1 as well, that replaced old nodes won't get cleaned up, generating large error logs. On CoreOS installation, I did systemctl restart kube-controller-manager. That fixed it.
Ping
This is happening on k8s 1.4.12 as well, restarting kube-controller-manager does not seem to fix it for me. It is rotating through two nodes that no longer exist, so I am hoping the cache will purge itself at some point and they will stop?
Just following up that the old nodes still not cleaned up
will work on a fix for this.
Let's make sure the fix also gets ported back to all affected branches (1.6, 1.5, and 1.4). CC
I was able to reproduce the problem by force deleting a node containing a pod with some volume attached to it. Did anyone run into a different failure mode?
I think that's how the bug surfaced for me. When I force killed my node we had just started running a StatefulSet with an attached persistent volume, and I think there was one running on the very node I killed.
The NodeInformer cache is actually reporting the correct state.

The problem is that inside the node status updater, there is another data structure keeping track of nodes that require a status update (see ActualStateOfWorld.GetVolumesToReportAttached(), called inside UpdateNodeStatuses() in ). The entry for the deleted node is never removed from the data structure; rather, the updater is set to try to update the dead node again next time, which leads to the message being logged every 100ms. The solution is to remove the corresponding node entry once the node is deleted. A fix is on its way.
A fix for 1.5 is on the way (PR )
Will 1.4 be included as well?
Here you go, my apologies.
Thank you very much! :)
To summarize, the fix for this is in 1.7 branch Releases 1.7.0 and greater 1.6 branch Releases 1.6.5 and greater 1.5 branch Releases 1.5.8 and greater (1.5.8 has not yet been cut, CC for ETA of that release). 1.4 branch Releases 1.4.13 and greater (1.4.13 has not yet been cut, CC for ETA of that release).
There is an edge case that the above fix did not handle: see PR is fixing that


Question: "strings" "testing" <ins> metainternalversion "k8s.io/apimachinery/pkg/apis/meta/internalversion" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" </ins> "net/http" "net/url" "time"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: tried to get listing of all componentstatuses using a labelSelector query selector. I am hitting the apiServer directly. GET /api/v1/componentstatuses What you expected to happen: GET /api/v1/componentstatuses with labelSelector should limit responses to only componentstatuses with the specified label How to reproduce it (as minimally and precisely as possible): run GET /api/v1/componentstatuses with and without a labelSelector against the apiServer. Anything else we need to know?: specifically my label selector is in the form . and i can confirm everything works as expected when i hit GET /api/v1/pods Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6+IKS", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-09T13:26:51Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: IBM OS (e.g: ): Mac Kernel (e.g. ): Darwin Kernel Version 18.5.0 Install tools: Using javascript npm request to talk to api server Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
i assume its because you cant add a label to componentstatus, but they still shouldnt return if i specify a labelSelector
that's indeed a bug i presume. but what use-case makes you label-querying componentstatus?
we have a bot to periodically run a query that finds all known endpoints on the ApiServer. Then for each of the found endpoints, find all resources with our label defined. sort of an inventory tracking system.
in a similar vein, the endpoint doesnt seem to respect
/priority backlog /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /priority backlog /help Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will take a look at this unless no objections
thank you, yours!
a PR for the ComponentStatus API, could you pls kindly take a look? I will look into the metrics API probably over the weekend. Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }) })) <ins> var _ = sigDescribe(feature.Windows, "SecurityContext", skipUnlessWindows(func() { f := framework.NewDefaultFramework("windows-with-unsupported-fields") f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged ginkgo.It("should be able to create pod and run containers", func(ctx context.Context) { ginkgo.By("Creating 1 pods: run with unsupported fields") pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "run-ignore-unsupported-fields", Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ NodeSelector: map[string]string{"kubernetes.io/os": "windows"}, Containers: []v1.Container{ { Name:  "test-container", Image: imageutils.GetE2EImage(imageutils.Pause), }, }, SecurityContext: &v1.PodSecurityContext{ RunAsUser:    ptr.To[int64](999), // windows does not support RunAsGroup:   ptr.To[int64](999), // windows does not support RunAsNonRoot: ptr.To(true), }, RestartPolicy: v1.RestartPolicyNever, }, } pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(ctx, pod, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error creating pod") podErr := e2epod.WaitForPodRunningInNamespace(ctx, f.ClientSet, pod) // Get the logs and events before calling ExpectNoError, so we can debug any errors. var logs string var events *v1.EventList if err := wait.PollUntilContextTimeout(ctx, 30*time.Second, 2*time.Minute, true, func(ctx context.Context) (done bool, err error) { framework.Logf("polling logs") logs, err = e2epod.GetPodLogs(ctx, f.ClientSet, f.Namespace.Name, pod.Name, pod.Spec.Containers[0].Name) if err != nil { framework.Logf("Error pulling logs: %v", err) return false, nil } events, err = f.ClientSet.CoreV1().Events(pod.Namespace).Search(scheme.Scheme, pod) if err != nil { return false, fmt.Errorf("error in listing events: %w", err) } return true, nil }); err != nil { framework.Failf("Unexpected error getting pod logs/events: %v", err) } else { framework.Logf("Pod logs: n%v", logs) framework.Logf("Pod events: n%v", format.Object(events, 1)) } framework.ExpectNoError(podErr) }) })) </ins> func runAsUserNamePod(username *string) *v1.Pod { podName := "run-as-username-" + string(uuid.NewUUID()) return &v1.Pod{

Install a cluster with v1.29.4 with linux node as master and worker node as windows kubeadm upgrade apply v1.30.0 fails with --ignore-preflight-errors` Describe Pod events: It seems that previously the pod was Pending as well, but this was ignored, because the jobs was successfully deleted in defer and return value was overridden with nil. Upgrade health check job expected to be scheduled only on Linux nodes using node selectors, rather than on Windows nodes. See What happened? No response_ <detailskubeadm version (use kubeadm version): v1.30.0 Environment: Kubernetes version (use kubectl version): v1.30.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 22.04.2 LTS, Windows Server 2022 Datacenter Kernel (e.g. uname -a): 5.19.0-1025-aws Container networking plugin (CNI): calico </details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in 1.30.1 /close
Closing this issue. <detailsIn response to : kubeadm version (use kubeadm version): v1.30.0 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubeadm
we are seeing this issue for 1.30.1 version too with upgrade health check pods getting scheduled on windows worker nodes.
looks like I was mistaken and this issue is different. but the health check uses the pause image, which is multi os, supposedly. the error is strange. can you show the full log output. we might have to apply a node selector on the job spec and backport to 1.31 another bug fix.

I'm not sure if we should add a node selector on job spec. the job uses the pause image to create a container. It doesn't have specific requirements for the node. After I look at the projected volume plugin, I'm confused why the above event occurs. On windows nodes, the function is empty. It doesn't do anything. On linux nodes, the function will set the owner of the volume. FYI: If a log file from the kubelet is provided, it will be helpful to understand the issue.
thanks for checking I also think there should not be an error. maybe we are hitting a core kubelet bug. is the kubelet on the windows node also 1.30?
please ignore the above comment, it's incorrect. I may find the root cause. It's a bug in kubelet when kubelet is running on Windows and the pod configures the SecurityContext with a RunAsUser.
/sig storage /sig windows
/assign I'm trying to fix the kubelet bug.
so this is not a kubeadm bug and we can move it back to the k/k repo?
/transfer kubernetes
/retitle kubelet on Windows fails if a pod has SecurityContext with RunAsUser.
Because of historical reasons, the and in is only designed for the Linux, not Windows. See for more details. adds a new field to the PodSpec, which can be used to identify the OS of the pod. It's an optional field and no default value. If unset, the above fields are not restricted by the api-server. In other words, the following job is allowed to be created. But it will fails to start on Windows nodes. And the root cause is that the is called by the . To fix this issue, the AtomicWriter can check the current OS of the node and skip the if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. I'm not sure if the failing job is expected or not. If it's expected, kubeadm has to change the job manifest to make it work on only Linux nodes. Why I ask this question is that I saw , which is used to set the ownership of the volume, if the os is Windows, it will become a no-op.

/cc
/cc
/triage accepted
slack discussion is happening here: but we have agreed that this is a kubelet problem. on the kubeadm side you can use the --ignore-preflight-errors=CreateJob option to skip the check. unclear when or if a kubelet fix will be and to what versions it will be available.
To fix this issue, the AtomicWriter can check the current OS of the node and skip the os.Chown if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. please see: do you want to upstream your carry?
thank you. I raise a PR and add as a co-author. I add an e2e test. do you know which job can test windows?
is the e2e test job for Windows


Question: dc.sync(ctx, pdbName) ps.VerifyDisruptionAllowed(t, pdbName, 0) <ins> verifyUnsupportedControllerEventEmitted(t, dc) </ins> } // Verify that an integer MaxUnavailable will recompute allowed disruptions when the scale of

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: glog.Errorf("Could not create volume unmounter for %s: %v", volume.Name, err) continue } <del> if unmounter == nil { glog.Errorf("Could not create volume unmounter for %s: %v", volume.Name, errUnsupportedVolumeType) continue } </del> tuple := cleanerTuple{Unmounter: unmounter} detacher, err := kl.newVolumeDetacherFromPlugins(volume.Kind, volume.Name, podUID)

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?