Question: PluginIsAttachable: true, // this field determines whether the plugin is attachable ReportedInUse:      true, } <del> oe.MountVolume(0 /* waitForAttachTimeout */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */) </del> <ins> oe.MountVolume(0 /* waitForAttachTimeout */, volumesToMount[i], nil /* actualStateOfWorldMounterUpdater */, false /* isRemount */) </ins> } // Assert

TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet.

The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks!

Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command.

This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!

Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.

because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days:


Question: "k8s.io/kubernetes/pkg/controller" "github.com/golang/glog" <ins> "github.com/juju/ratelimit" </ins> ) type CertificateController struct {

Each node waiting for bootstrapping creates 1 line per 2-3s in the logs when the requesting client doesn't have access. It should not be logging at v(0), these are all v(4) level. We should be reporting these appropriately. I don't want a 5k node cluster showing this.
I'd like to fix it


Question: return nil } <del> // Verify that the profile is valid and loaded. func validateProfile(profile string, loadedProfiles map[string]bool) error { if err := ValidateProfileFormat(profile); err != nil { return err } if strings.HasPrefix(profile, v1.AppArmorBetaProfileNamePrefix) { profileName := strings.TrimPrefix(profile, v1.AppArmorBetaProfileNamePrefix) if !loadedProfiles[profileName] { return fmt.Errorf("profile %q is not loaded", profileName) } } return nil } </del> // ValidateProfileFormat checks the format of the profile. func ValidateProfileFormat(profile string) error { if profile == "" || profile == v1.AppArmorBetaProfileRuntimeDefault || profile == v1.AppArmorBetaProfileNameUnconfined {

What happened: Pods with invalid container-level seccomp profiles allow its init containers to be executed. Pods with unloaded apparmor profiles blocks the entire pod. What you expected to happen: Consistency between both. In one hand AppArmor blocks all pod execution regardless of the containers having or not a profile attached to it. On the other, Seccomp allows for an init container to deploy its own profile - as currently done on . How to reproduce it (as minimally and precisely as possible): Apply the yaml below: Resulting in only the init container from the being executed: ! Anything else we need to know?: AppArmor is currently being target to GA on 1.21 (). Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.0-alpha.0.292+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-12T23:13:21Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /area security
/cc
/sig security /triage accepted
I think we can consider this as a bug, right? Then we could work on a fix for v1.21 interdependently of the AppArmor GA graduation.
Sounds like a good plan :+1:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_HTTPIngressPath = map[string]string{ "":         "HTTPIngressPath associates a path with a backend. Incoming urls matching the path are forwarded to the backend.", <del> "path":     "Path is matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/'. When unspecified, all paths from incoming requests are matched.", </del> <ins> "path":     "Path is matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/' and must be present when using PathType with value "Exact" or "Prefix".", </ins> "pathType": "PathType determines the interpretation of the Path matching. PathType can be one of the following values: * Exact: Matches the URL path exactly. * Prefix: Matches based on a URL path prefix split by '/'. Matching isn  done on a path element by element basis. A path element refers is then  list of labels in the path split by the '/' separator. A request is an  match for path p if every p is an element-wise prefix of p of then  request path. Note that if the last element of the path is a substringn  of the last element in request path, it is not a match (e.g. /foo/barn  matches /foo/bar/baz, but does not match /foo/barbaz).n* ImplementationSpecific: Interpretation of the Path matching is up ton  the IngressClass. Implementations can treat this as a separate PathTypen  or treat it identically to Prefix or Exact path types.nImplementations are required to support all path types.", "backend":  "Backend defines the referenced service endpoint to which the traffic will be forwarded to.", }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I tried to use the default path I got . No error. According to the path is not needed. "When unspecified, all paths from incoming requests are matched." Create a ingress like so: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: brew Network plugin and version (if this is a network-related bug): N/A Others:
/sig network /area ingress It seems to be a problem of ingress and I suggest creating an issue in I find a similar one which is a little different there
We use - would this be a problem with the aws repo then?
/assign
So digging a bit here, the API docs needs to be improved. By the , is required if the PathPrefix is Exact or Prefix. To solve your issue specifically, you have two ways: Add that accepts ingress without paths Add and change PathPrefix to Prefix, so this becomes a catchAll I'll open a PR to fix the comment/docs. Thanks!
Thanks! In my case because I'm using ALB controller I had to use with prefix to get it working - see
/triage accepted PR will close this ;)


Question: statuses = make([]autoscalingv2.MetricStatus, len(metricSpecs)) <ins> if scale.Status.Selector == "" { errMsg := "selector is required" a.eventRecorder.Event(hpa, v1.EventTypeWarning, "SelectorRequired", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", "the HPA target's scale is missing a selector") return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } selector, err := labels.Parse(scale.Status.Selector) if err != nil { errMsg := fmt.Sprintf("couldn't convert selector into a corresponding internal selector object: %v", err) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidSelector", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", errMsg) return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } invalidMetricsCount := 0 var invalidMetricError error </ins> for i, metricSpec := range metricSpecs { <del> if scale.Status.Selector == "" { errMsg := "selector is required" a.eventRecorder.Event(hpa, v1.EventTypeWarning, "SelectorRequired", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", "the HPA target's scale is missing a selector") return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } </del> <ins> replicaCountProposal, metricNameProposal, timestampProposal, err := a.computeReplicasForMetric(hpa, metricSpec, currentReplicas, selector, &statuses[i]) </ins> <del> selector, err := labels.Parse(scale.Status.Selector) </del> if err != nil { <del> errMsg := fmt.Sprintf("couldn't convert selector into a corresponding internal selector object: %v", err) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidSelector", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidSelector", errMsg) return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) } var replicaCountProposal int32 var timestampProposal time.Time var metricNameProposal string switch metricSpec.Type { case autoscalingv2.ObjectMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(metricSpec.Object.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetObjectMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetObjectMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", nil, time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForObjectMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i], metricSelector) if err != nil { return 0, "", nil, time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.PodsMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(metricSpec.Pods.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetPodsMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetPodsMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", nil, time.Time{}, fmt.Errorf("failed to get pods metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForPodsMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i], metricSelector) if err != nil { return 0, "", nil, time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.ResourceMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForResourceMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i]) if err != nil { return 0, "", nil, time.Time{}, err } case autoscalingv2.ExternalMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForExternalMetric(currentReplicas, metricSpec, hpa, selector, &statuses[i]) if err != nil { return 0, "", nil, time.Time{}, err } default: errMsg := fmt.Sprintf("unknown metric source type %q", string(metricSpec.Type)) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidMetricSourceType", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidMetricSourceType", "the HPA was unable to compute the replica count: %s", errMsg) return 0, "", nil, time.Time{}, fmt.Errorf(errMsg) </del> <ins> invalidMetricsCount++ invalidMetricError = err </ins> } <del> if replicas == 0 || replicaCountProposal > replicas { </del> <ins> if err == nil && (replicas == 0 || replicaCountProposal > replicas) { </ins> timestamp = timestampProposal replicas = replicaCountProposal metric = metricNameProposal } } <del> setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionTrue, "ValidMetricFound", "the HPA was able to successfully calculate a replica count from %s", metric) </del> <ins> // If all metrics are invalid or some are invalid and we would scale down, // return an error and set the condition of the hpa based on the first invalid metric. // Otherwise set the condition as scaling active as we're going to scale if invalidMetricsCount >= len(metricSpecs) || (invalidMetricsCount > 0 && replicas < currentReplicas) { return 0, "", statuses, time.Time{}, fmt.Errorf("Invalid metrics (%v invalid out of %v), last error was: %v", invalidMetricsCount, len(metricSpecs), invalidMetricError) } else { setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionTrue, "ValidMetricFound", "the HPA was able to successfully calculate a replica count from %v", metric) } </ins> return replicas, metric, statuses, timestamp, nil } <ins> // computeReplicasForMetric computes the desired number of replicas for for a specific hpa and single metric specification. func (a *HorizontalController) computeReplicasForMetric(hpa *autoscalingv2.HorizontalPodAutoscaler, spec autoscalingv2.MetricSpec, currentReplicas int32, selector labels.Selector, status *autoscalingv2.MetricStatus) (replicaCountProposal int32, metricNameProposal string, timestampProposal time.Time, err error) { switch spec.Type { case autoscalingv2.ObjectMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(spec.Object.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetObjectMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetObjectMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForObjectMetric(currentReplicas, spec, hpa, selector, status, metricSelector) if err != nil { return 0, "", time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.PodsMetricSourceType: metricSelector, err := metav1.LabelSelectorAsSelector(spec.Pods.Metric.Selector) if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetPodsMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetPodsMetric", "the HPA was unable to compute the replica count: %v", err) return 0, "", time.Time{}, fmt.Errorf("failed to get pods metric value: %v", err) } replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForPodsMetric(currentReplicas, spec, hpa, selector, status, metricSelector) if err != nil { return 0, "", time.Time{}, fmt.Errorf("failed to get object metric value: %v", err) } case autoscalingv2.ResourceMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForResourceMetric(currentReplicas, spec, hpa, selector, status) if err != nil { return 0, "", time.Time{}, err } case autoscalingv2.ExternalMetricSourceType: replicaCountProposal, timestampProposal, metricNameProposal, err = a.computeStatusForExternalMetric(currentReplicas, spec, hpa, selector, status) if err != nil { return 0, "", time.Time{}, err } default: errMsg := fmt.Sprintf("unknown metric source type %q", string(spec.Type)) a.eventRecorder.Event(hpa, v1.EventTypeWarning, "InvalidMetricSourceType", errMsg) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "InvalidMetricSourceType", "the HPA was unable to compute the replica count: %v", fmt.Errorf(errMsg)) return 0, "", time.Time{}, fmt.Errorf(errMsg) } return replicaCountProposal, metricNameProposal, timestampProposal, nil } </ins> func (a *HorizontalController) reconcileKey(key string) (deleted bool, err error) { namespace, name, err := cache.SplitMetaNamespaceKey(key) if err != nil {

What happened: I setup custom metrics based autoscaling with multiple metrics. One of those metrics was not available. HPA reports all metrics as "unknown" (even the ones that are still available) and refuses to operate. What you expected to happen: Pods would be automatically scaled based on metrics that were available, ignoring missing metrics (treating them as zero). A partial failure in the metrics system should not prevent auto-scaling from proceeding with the data that it has. For example, CPU % metrics come from the k8s metrics server whereas the other metrics come from the prometheus adapter. If the prometheus adapter goes away we can still use the CPU metric as a lower bound on the number of replicas. If you consider, e.g. , instead of aborting the whole metric gathering process there, it could have stored a placeholder or zero metric value when it fails to fetch the metric. How to reproduce it (as minimally and precisely as possible): Setup a custom metrics server and tell HPA to scale based on several metrics, one of which is not actually available. Environment: kops 1.8 on AWS Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
/sig autoscaling
cc
/kind bug /priority important-longterm
this return returns nil value of statuses variable. statuses variable contains the value of all metrics.
as long as we failed to get one metric of all metrics, the func computeReplicasForMetrics returns, the scale lose efficacy, it is not highly available. when failed to get metric, should we record this event and do not return:)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still having this problem.
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this issue was resolved or it works as designed? thanks,
I was testing this last week and it is not resolved. I don't think it is reasonable to say this is working as designed.
Seems there is a PR to fix it, but it hasn't been reviewed or merged.
yes, I looked into the master branch code, not fixed yet, just curious about it, it would not be fixed or as dobesv said it will be merged/fixed latter with that PR.

/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue was discussed at SIG Autoscaling yesterday: I'm going to raise a new PR later this week given the code's changed so much since the original PR was raised by that it doesn't make sense to just rebase it.
Thanks, that would be excellent!


Question: execFn func(), ) <ins> // MaintainObservations is a helper for maintaining statistics. MaintainObservations(stopCh <-chan struct{}) </ins> // Run monitors config objects from the main apiservers and causes // any needed changes to local behavior.  This method ceases // activity and returns after the given channel is closed.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Not observed yet, but deduced from analyzing the code. If a k/apiserver/pkg/util/flowcontrol/fairqueuing.QueueSet, whose metrics.TimedObserverPair is a pair of k/apiserver/pkg/util/flowcontrol/metrics.sampleAndWaterMarkHistograms, gets zero activity for a long while and then gets some activity, then the loop at will go through a lot of iterations, and do so with a lock held. What you expected to happen: No big hiccups in any scenario. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing I noticed an incorrect fake implementation of Interface. I looked at the comment on the Handle method, and noticed that it does not say when/how the is called. What you expected to happen: I expected the parameter to be as well documented as the others. How to reproduce it (as minimally and precisely as possible): Look at and see how it does not say when/whether/with-what-parameter-values is called. Anything else we need to know?: Environment: Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Any chance we could write a test for the interface which verifies correct plumbing?
/cc
: that should be do-able


Question: } }) <del> ginkgo.It("verify PriorityClass endpoints can be operated with different HTTP methods", func() { </del> <ins> /* Release: v1.20 Testname: Scheduler, Verify PriorityClass endpoints Description: Verify that PriorityClass endpoints can be listed. When any mutable field is either patched or updated it MUST succeed. When any immutable field is either patched or updated it MUST fail. */ framework.ConformanceIt("verify PriorityClass endpoints can be operated with different HTTP methods", func() { </ins> // 1. Patch/Update on immutable fields will fail. pcCopy := pcs[0].DeepCopy() pcCopy.Value = pcCopy.Value * 10

The following PriorityClass endpoints was Promoted it 1.14 without Conformance tests, and remain untested: replace-, read-, patch-, list-, delete-collection- SchedulingV1PriorityClass According to this APIsnoop query, there are still some remaining PriorityClass endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints ! createSchedulingV1PriorityClass deleteSchedulingV1PriorityClass Promoted to Conformance 1.19 with /sig testing /sig architecture /area conformance
Is the ultimiate goal to use conformance tests to explicitly issue all CRUD requests against PriorityClass? And, is equivalent to "update PriorityClass"?
all APIs must have Conformance tests unless they are not eligible for Conformance due to the There is other APIGroups that have both Replace and Update as endpoints. That is the prerogative of the SIG to decide.
can you please point us to similar conformance tests used for core objects such as pods and nodes?
This is a life cycle test for Create, List, Update, Patch, DeleteCollection More can be found in the test/e2e folder. Also look for examples on the Conformance at previously Promoted Conformance test Reach out if we can help with more information on our approach to Conformance testing.
I remembered about this PR. Last week a few lines of code was to cover the last of the "Lease" endpoint group for conformance. The one file contain the Conformance tests for the complete life cycle of Lease endpoints.
/assign I will work on it.
/reopen Will monitor in test grid to move to Conformance after 2 weeks soak.
Reopened this issue. <detailsIn response to : /reopen Will monitor in test grid to move to Conformance after 2 weeks soak. </details
/triage accepted


Question: ginkgo.By(fmt.Sprintf("deleting the test namespace: %s", ns1)) // Delete the primary namespace but its okay to fail here because this namespace will // also be deleted by framework.Aftereach hook <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns1, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns1) }) </ins> ginkgo.By("uninstalling csi mock driver") tryFunc(func() {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-ubuntu-gce-containerd Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Pre-provisioned PV (default fs)] subPath should support file as subpath [LinuxOnly] Testgrid link: Reason for failure: Will look at it shortly, why it is flaking. Anything else we need to know: links to appreciatedlinks to specific failures in spyglass appreciated
/sig storage
The namespace terminating during test setup is super strange but seems like a test-only issue, not an issue with the functionality being tested. I'd recommend looking at the test, API server, and controller logs to figure out why the test namespace () was deleted and who deleted it
does this help ?
feels like a test only problem like said. We don't even have gluster in our CI/prow to test against... cc Michelle, Xing, Can you please confirm that this is not a blocker for 1.19.0?
Knowing the timeframe for when the test started and when that delete happened would be interesting to know. Possibilities I can see: something in the test timed out, and the cleanup happened in parallel with something in the test attempting to run multiple tests are reusing the same test context and namespace name at the same time (I can never keep straight how ginkgo parallel tests work)
This does not look like a blocker for 1.19.0.
Thanks /milestone v1.20
looking at the logs it seems to be the latter "same test context and namespace name at the same time", but we can deal with this for 1.19.1
cc This looks like another instance of "the same test namespace is being reused". I believe had a mitigation for that to try to make the namespaces more unique.
(btw, the test brings up a containerized gluster server so we are exercising gluster functionality)
Yeah, looking at the build log found that it is the same issue as The issue is still around the double deletion of namespace, i.e. we delete the test namespace during the test execution, but later on the call we delete the namespace again to ensure it truly get cleaned up.

But unfortunately, this namespace has been reused by another test case between this gap.
Some potential fix I can think of: deletion of the primary namespace twice is inevitable, maybe we should consider generate primary namespace based on a hashing of the test cases name? the 4 digits random namespace to 6, this should not solve the problem in nature but it will extremely decrease the chance it happens. using an in-memory namespace map to keep record of already generated namespaces for the whole test run and refuse to generate the same namespace again in the whole lifecycle of a prow job.
do you know if this gluster test is using your mitigation in ?
hash of the testcase name + random suffix (because tests can be run multiple times) seems like a good way to mitigate the chances of tests reusing the same namespace. I'm unsure if we know the test case name at the time of test namespace creation though. I think increasing the length of the random suffix is a quick and easy mitigation believe ginkgo to run in parallel, so this may not work
Yes it is. But my fix in can only reduce the chance of collision and probably improve readability of logs. But not to eliminate this issue in nature.
AFAICT does not affect intree code at all, because it only fixes namespace of driver. If we are seeing collision of primary namespace for intree code then it is a different problem.
on only deals with the collision between test namespace and driver namespace. And that fix is taking affect. failure is because a test namespace collides with another test namespace. And this is not related to in-tree code at all. Any test could potentially become flaky because their namespace can be deleted during test execution. It's just happens to be this intree test this time. The root cause for this problem is: We cleaned up the test namespace twice which gives it a small gap that another testcase can generate the same namespace and be affected by the second namespace deletion from the original testcase.
/assign I would like to give it a shot.
I've opened to address this issue.
/unassign /assign


Question: f := input.f skipExecTest(input.resource.driver) <del> testScriptInPod(f, input.resource.volType, input.resource.volSource) </del> <ins> testScriptInPod(f, input.resource.volType, input.resource.volSource, input.config.NodeSelector) </ins> }) } <del> func testScriptInPod(f *framework.Framework, volumeType string, source *v1.VolumeSource) { </del> <ins> func testScriptInPod( f *framework.Framework, volumeType string, source *v1.VolumeSource, nodeSelector map[string]string) { </ins> const ( volPath = "/vol1" volName = "vol1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Test pod is not created successfully, likely due to it being scheduled in a different zone from where the PD is provisioned. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
/sig storage /assign /priority critical-urgent


Question: framework.Skipf("Runs only when number of nodes >= 2") } <del> ep := &eventPatterns{ reason:  "FailedMount", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "NodeSelectorTerm") ep.pattern = append(ep.pattern, "MountVolume.NodeAffinity check failed") </del> testVols := setupLocalVolumesPVCsPVs(config, DirectoryLocalVolumeType, config.node0, 1, immediateMode) testVol := testVols[0]

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Based on discussion , events can be throttled and should not be used to validate failure conditions in e2e tests. We should remove the event checks from subpath and local pv tests and just wait for pod failure. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
could you help look at this for local PV tests?
sure /assign
/area deflake
"Invalid AWS KMS key" also waits for an event

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup tests are flaking 1-2 times a day: https://k8s- Some test logs: https://k8s- https://k8s-
could you take a look?
I'll look into this.
I found why the tests failed, see my analysis of logs: I created a PR () trying to fix the issue. I am still investigating kubelet and trying to reproduce the issue locally.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen


Question: var _ Interface = new(cadvisorUnsupported) <del> func New(address string, port uint, imageFsInfoProvider ImageFsInfoProvider, rootPath string) (Interface, error) { </del> <ins> func New(address string, port uint, imageFsInfoProvider ImageFsInfoProvider, rootPath string, usingLegacyStats bool) (Interface, error) { </ins> return &cadvisorUnsupported{}, nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is failing with: What you expected to happen: The cross build should work,. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It looks like this happened in , probably because we don't block the queue on cross :-( xref: /priority failing-test /sig release
Additionally ideally sig-release should for this job.
/assign
, is making a fix and then we'll look at making sure cross blocks PRs it runs against in the future.


Question: requiresUpdate := kl.reconcileCMADAnnotationWithExistingNode(node, existingNode) requiresUpdate = kl.updateDefaultLabels(node, existingNode) || requiresUpdate requiresUpdate = kl.reconcileExtendedResource(node, existingNode) || requiresUpdate <ins> requiresUpdate = kl.reconcileHugePageResource(node, existingNode) || requiresUpdate </ins> if requiresUpdate { if _, _, err := nodeutil.PatchNodeStatus(kl.kubeClient.CoreV1(), types.NodeName(kl.nodeName), originalNode, existingNode); err != nil { klog.Errorf("Unable to reconcile node %q with API server: error updating node: %v", kl.nodeName, err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files.

I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?


Question: to delete the secret, the deletion must succeed. release: v1.21 file: test/e2e/common/storage/secrets_volume.go <ins> - testname: StorageClass, lifecycle codename: '[sig-storage] StorageClasses CSI Conformance should run through the lifecycle of a StorageClass [Conformance]' description: Creating a StorageClass MUST succeed. Reading the StorageClass MUST succeed. Patching the StorageClass MUST succeed with its new label found. Deleting the StorageClass MUST succeed and it MUST be confirmed. Replacement StorageClass MUST be created. Updating the StorageClass MUST succeed with its new label found. Deleting the StorageClass via deleteCollection MUST succeed and it MUST be confirmed. release: v1.29 file: test/e2e/storage/storageclass.go </ins> - testname: 'SubPath: Reading content from a configmap volume.' codename: '[sig-storage] Subpath Atomic writer volumes should support subpaths with configmap pod [Conformance]'

[X] test approval issue : [X] test pr : [X] two weeks soak end date : 22 Sept 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three StorageClass endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following StorageClass endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/sig storage
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: FilenameOptions resource.FilenameOptions ServerSideApply bool <ins> FieldManager    string </ins> ForceConflicts  bool OpenAPISchema    openapi.Resources

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.


Question: return service } <del> func createLBServiceOrFail(clientset *fedclientset.Clientset, namespace, name string) *v1.Service { service, err := createLBService(clientset, namespace, name) </del> <ins> func createLBServiceOrFail(clientset *fedclientset.Clientset, namespace, name string, clusters fedframework.ClusterSlice) *v1.Service { service, err := createLBService(clientset, namespace, name, clusters) </ins> framework.ExpectNoError(err, "Creating service %q in namespace %q", service.Name, namespace) By(fmt.Sprintf("Successfully created federated service (type: load balancer) %q in namespace %q", name, namespace)) return service

In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -


Question: errorType:   field.ErrorTypeInvalid, errorDetail: "must be a valid seccomp profile", }, <ins> "invalid seccomp allow any default profile": { psp:         invalidSeccompAllowAnyDefault, errorType:   field.ErrorTypeInvalid, errorDetail: "must be a valid seccomp profile", }, </ins> "invalid seccomp allowed profile": { psp:         invalidSeccompAllowed, errorType:   field.ErrorTypeInvalid,

It should be possible to create a PodSecurityPolicy with zero restrictions. This was the intent with the (a.k.a. "AllowAny") allowed seccomp profile, but it is rejected during the validation phase. /kind bug
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/auth: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: import ( "fmt" <ins> "strings" </ins> "testing" <ins> "github.com/google/go-cmp/cmp" "github.com/stretchr/testify/assert" </ins> ) func TestThreadSafeStoreDeleteRemovesEmptySetsFromIndex(t *testing.T) {

This was the root cause of rancher/rancher. In , when is called, it is given the oldVersion of an object and the newVersion of the object. It then uses the indexers to calculate the old values (from the old version) and the new updated values. In cases where are size of 1, are of size greater than 1, and contain the sole value from , the key will be deleted for the old value will be deleted and won't be re-. This is due to the two for loops at the end of the method which are used for optimization. When the first for loop is processed, . Because of this, the key is deleted from the index for this value. In the second for loop, the , and for at least one of the values . The continue statement causes the key to not be to the index for this value (and since the value was deleted in the prior for loop, it no longer exists in the index for this value), despite the fact that the indicated that this is the desired state. Since specified that this value was still valid for this object, the key should not be deleted from the index for this value, or it should be re- when it is re-discovered in . Either way, the end state of the index should include this key in the set for the specified value. I've created a which demonstrates the issue (branch is ). To use it (assuming that you have a running kubernetes cluster to use with): the controller: . the foo crd: the controller (assumes that you have $KUBECONFIG set to the file path of your kubeconfig file). an example foo with only one label: . Observe, in the logs of the running sample-controller, that the indexer was able to locate this foo using the indexer. the foo to include a second label: . Observe, in the logs of the running sample-controller, that the indexer was not able to locate the foo using the first key any longer, despite the fact that the label was still on the object. The sample-controller fork adds an indexer for foo objects which uses the indexerFunction of . This function looks for labels that have a key with a prefix of and uses their values as index keys for the foo objects. For example, if I have a with a label of , the indexer will allow you to retrieve using .

The controller's method has been altered to just ensure that the foos can be retrieved using the indexer for all valid labels. If the controller can't retrieve foo by using the value from a valid label, then it returns an error (it tries for all valid labels). This was originally discovered in a fork of client-go that we maintain. After we reverted the changes introduced by , our issues stopped occurring. N/A - This was introduced in and appears to be in all versions of the client v0.23+. N/A - issue is with client-go No response No response No response No response
/sig api-machinery
I think that you have background in this issue/area of the code. Would you mind taking a look at this issue and the PR that I have up to fix it? Please let me know if I've misunderstood anything.
/assign Would you mind to take a look? Thank you /triage accepted
Thanks for reporting the issue. will address this issue
Reopening - since we need to cherrypick back to 1.23
Cherrypick is merged - closing.


Question: importpath = "k8s.io/kubernetes/pkg/kubelet/eviction", deps = [ "//pkg/api/v1/resource:go_default_library", <ins> "//pkg/apis/core/v1/helper:go_default_library", </ins> "//pkg/apis/core/v1/helper/qos:go_default_library", "//pkg/features:go_default_library", "//pkg/kubelet/apis/stats/v1alpha1:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug Description: According to the preemption's doc, we'll take priority over Qos; but in , the higher priority BestEffort pod maybe rejected because of memory pressure. If TaintNodesByCondition enabled, the pod need memory pressure toleration. /cc , correct me if any mis-understanding.
/sig scheduling /sig node
If the node has memory pressure, it rejects all pods that do not tolerate it. This is orthogonal to QoS and Priority.
Preemption and node evictions do not work with the same exact metrics. Preemption is unaware of usage, while node eviction logic has pod and node resource usage information. Node eviction takes usage into account both when admitting a new pod and when evicting an existing pod. has recently sent out to update node eviction doc.
, but will also reject it if pod tolerate it :(
, please refer to for what I mean :). BTW, you need to configure out and predicates, and enable alpha feature. /cc
I'm not familiar with the Preemption/Priority design, so I don't know where is preemption happening (Node or control plane). But despite that I wouldn't read what you wrote as inconsistency. Priority takes precedence over QoS when it can. But if some QoS Pods are not allowed to run somewhere it's not an issue with preemption/priority or QoS. It's a separate predicate. Slight OT, but related to your proposal, I think we should go another way. We should prevent BE pods to get MemoryPressure toleration. Having this can result with bad user experience, as Pod will get scheduled on a Node, and then immediately killed, because Kubelet ignores tolerations. Or we can teach kubelet respect tolerations for NC created taints. I'd consider it a blocker for beta of TaintNodeByCondition.
Yes; but if BE pod has higher priority, the other pods with lower priority will be evicted by kubelet. This maybe better, no idea on which pod will be BE but with higher priority :) , do you think we should reject BE pod with MemoryPressure toleration, even it has a higher priority?

I would like to mention several points here: of all, I should say that a high priority BE pod is a result of bad configuration and we should not optimize for it, but given that it is not rejected by the system, we must handle it. BE pod will never cause preemption due to lack of resources, because a BE pod needs zero resources from scheduler's point of view. of point , I think we should not worry about a high priority BE pod getting rejected by kubelet or causing other pods to be evicted from a node. We should document the system behavior and have clear guidelines and point out the problems that could be caused by bad configurations. After that we should leave it to the user to decide if a configuration makes sense for their use-case.
Overall, I agree :). Personally, I'd like to keep components behavior align: reject by admission/kubelet or accept by them. Currently, apiserver/scheduler accept and handle it and kubelet will reject it.
Kubelet does not reject a BE pod with high priority either.
I mean BE pod with Memory Pressure toleration :)
I see. The pod may still be scheduled without any problem if another node in the cluster is found that does not have memory pressure. So, I wouldn't call this inconsistent behavior of components. Anyway, I leave it up to SIG node to decide whehter the current behavior should be changed.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: kl.updatePodCIDR(node.Spec.PodCIDR) <del> if err := kl.setNodeStatus(node); err != nil { return err } </del> <ins> kl.setNodeStatus(node) </ins> // Patch the current status on the API server updatedNode, err := nodeutil.PatchNodeStatus(kl.kubeClient, types.NodeName(kl.nodeName), originalNode, node) if err != nil {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.3.8 custom build - the code seems to still be in place in master. Environment: Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): VERSION="14.04.5 LTS, Trusty Tahr" Kernel (e.g. ): 4.4.0-38-generic ~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: Custom Others: What happened: When the OpenStack Nova API is not accessible Nodes are marked as "NotReady" The OpenStack cloud providers seems to call Nova on a schedule to get information. If the API is not contactable, kubelet fails to call back to the masters, and gets marked as NotReady. Masters also call the Nova API, and when it fails, it can mark all nodes as NotReady. What you expected to happen: Nothing - the cluster should still work, regardless of Nova's API How to reproduce it (as minimally and precisely as possible): Install Kubernetes using the OpenStack provider. On the OpenStack controller stop the nova-api process.


Question: }, { "ImportPath": "github.com/ghodss/yaml", <del> "Rev": "3bc1590d16074751993dd3b1a76e7a8d1a916a11" </del> <ins> "Rev": "588cb435e59ee8b6c2795482887755841ad67207" </ins> }, { "ImportPath": "github.com/golang/glog",

I spin up a new cluster using GKE at version 0.9.2. Then I run: $kubectl get minion <minion-id-o yaml This returns a yaml file: apiVersion: v1beta1 creationTimestamp: 2015-02-03T01:16:46Z hostIP: <ipid: <minion-idkind: Node labels: diskSize: "massive" resourceVersion: 12 resources: capacity: cpu: "1" memory: 4.+09 selfLink: /api/v1beta1/minions/<minion-idstatus: conditions: kind: Ready lastTransitionTime: null status: Full uid: -ab42-11e4-9817- I delete the hostIp and status so that the update request won't get rejected ( should fix that), then run: $ kubectl update -f ERROR: F0202 18:08:19. 767 ] error unmarshaling JSON: json: cannot unmarshal number 4.+09 into Go value of type int I'm not sure whether the marshaling or the unmarshaling is what needs to be fixed, but the API shouldn't be returning numbers that it can't parse.
It looks like Quantity is unmarshalling wrong? , Alex Robinson wrote:
That's very likely. And sorry, I didn't realize yaml didn't paste into comments well. Here's the actual yaml:
Yeah, on the client-side, apparently, since the error is in It's possible that was missed during the Quantity refactoring. It may be troublesome that the v1beta1 API accepts a util.IntOrString for resources rather than a Quantity:
/cc
The (external) yaml library that we use is the cause for this. I will send a patch to fix the yaml library.
Sent a to our yaml dependency.
If we are only fixing our yaml unmarshalling, we are still losing precision. E.g. The yaml that Alex posted above is supposed to be representing 3.75 GiB. 75 GiB * 2^30 = , which we are representing as 4.+09. The amount of memory we are "losing" here is trivial, but if this is actually using scientific notation, we could be rounding up half the time as well (creating a very very low chance of an unexpected OOM).

Both of these are pretty small problems, but it seems silly to needlessly lose precision. Can we just not write these in scientific notation?
Resource quantities must be lossless. , CJ Cullen wrote:
+1. These are supposed to be strings, not integers nor floats.
What does return?
That does look like an API bug. It should be a string, IIUC.
Legacy CPU and memory are int and int64, respectively. Resource.Quantity should always produce strings.
This can't possibly work. IntOrString is a 32-bit signed int. This value doesn't fit in 32 bits.
Yes - this isn't a problem with the YAML library. This is a problem with trying to render a number larger than 32 bits () as a 32-bit integer (IntOrString), which instead ends up rendering as a floating point. (or something that keeps this value as a 64-bit integer the whole time) is the right fix.
The yaml library converts an into a as part of Marshal. Isn't that an issue with the yaml library? I agree that the kubernetes API has another issue which is that of not being able to represent resources greater than MaxInt. But to me these are two separate issues.
Even an int32 gets marshaled into a floating point number. I don't see how that is valid. The underlying golang package has separate APIs to address this behavior and the YAML library will also have to do that.
IntOrString stores things as "int", which is 32 or 64 bits depending on the platform. We should explicitly use int32 or int64 in our api objects. These values should be stored as a Quantity in v1beta3, and converted to a string in v1beta1 & 2.
Ack , Daniel Smith wrote:
If someone changed IntOrString to use int64 I would approve. , Vish Kannan wrote:
Assigned to myself.


Question: err = pd.mounter.Mount(globalPDPath, pd.GetPath(), "", mount.FlagBind|flags, "") if err != nil { os.RemoveAll(pd.GetPath()) <ins> // TODO: we should really eject the attach/detach out into its own control loop. detachDiskLogError(pd) </ins> return err }

Repro: Create a pod that attempts to mount a GCE PD that is mis-formatted Delete the pod Expected: GCE PD should be detached Observed: GCE PD is not detached.
Chatted with offline and understand the problem with more detail: The basic problem is when filesystem on PD is unformatted, the mount is going to fail. Kubernetes doesn't do cleanup by detaching the PD. Two potential solutions were proposed: 1) use built-in utility: safeformatand_mount in GCE image to format and mount PD to avoid extra step to format the PD disk. 2) Handle mount failure properly. I think we might need handle both anyway. Brendan mentioned that you are looking at a potential new controller for PD, whose responsible is attaching and detaching PD disks. Do you have any latest status on this topic?
No latest status. I haven't done any actual work on it yet. The idea would be to scan the pods assigned to each node, and attaches (or detaches) any PDs that the node needs (or no longer needs). I'm still coming up to speed on a lot of the architectural concepts, so I'd love some guidance.
Now that Tim's volume re-factor has gone in, I'm going to re-visit this issue. Assigning to me.


Question: }, } <ins> fqinSecret := &corev1.Secret{ ObjectMeta: metaObj, Data: map[string][]byte{ testServerFQIN + "." + userKey:     []byte(testUser), testServerFQIN + "." + passwordKey: []byte(testPassword), }, } </ins> emptySecret := &corev1.Secret{ ObjectMeta: metaObj, Data:       map[string][]byte{},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-controller-manager is unable to locate the credentials of the vCenter host, even though they are defined and cased identically to the hostname in the cloud-provider-config. What you expected to happen: vCenter hostname should not be case-sensitive, or should be documented that it is. It should successfully locate credentials from the manager How to reproduce it (as minimally and precisely as possible): to an upper-case name kube-controller-manager daemon to update the secret to provision storage with the "thin" storage class will stay suck in pending Anything else we need to know?: Log Messages: ` This appears to be induced by the way the vCenter server name is calculated when the secret containing the credentials is parsed[]. The credentialKey is converted to lower case and is then parsed to lookup the server name. Environment: Kubernetes version (use ): v1.14 Cloud provider or hardware configuration: vSphere OS (e.g: ): Unknown Kernel (e.g. ): Unknown Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area/provider/vmware
/area provider/vmware


Question: return errStr + buf.String() } <del> // check if a Pod is controlled by a Replication Controller in the List func hasReplicationControllersForPod(rcs *api.ReplicationControllerList, pod api.Pod) bool { for _, rc := range rcs.Items { selector := labels.SelectorFromSet(rc.Spec.Selector) if selector.Matches(labels.Set(pod.ObjectMeta.Labels)) { return true } } return false } </del> // WaitForPodsSuccess waits till all labels matching the given selector enter // the Success state. The caller is expected to only invoke this method once the // pods have been created.

The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc


Question: file: test/e2e/apimachinery/validatingadmissionpolicy.go - testname: ValidatingAdmissionPolicy codename: '[sig-api-machinery] ValidatingAdmissionPolicy [Privileged:ClusterAdmin] <del> should type check a CRD [Conformance]' description: ' The ValidatingAdmissionPolicy should type check a CRD.' release: v1.30 file: test/e2e/apimachinery/validatingadmissionpolicy.go - testname: ValidatingAdmissionPolicy codename: '[sig-api-machinery] ValidatingAdmissionPolicy [Privileged:ClusterAdmin] should type check validation expressions [Conformance]' description: ' The ValidatingAdmissionPolicy should type check the expressions defined inside policy.' release: v1.30 file: test/e2e/apimachinery/validatingadmissionpolicy.go - testname: ValidatingAdmissionPolicy codename: '[sig-api-machinery] ValidatingAdmissionPolicy [Privileged:ClusterAdmin] </del> should validate against a Deployment [Conformance]' description: ' The ValidatingAdmissionPolicy should validate a deployment as the expression defined inside the policy.'

<img width="1015" alt="image" src="" /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig api-machinery
/priority critical-urgent Looking at the last green and first red shows the following: xref : /assign
hrmm, looks super solid in
yea something screwy going on with the kubeadm jobs :(
my main concern what that this was a test ..
kinder or kind jobs have been mostly solid across the board for a few years. if something is failing in kind / kinder then likely it would also fail in resource constraint scenarios. sorry but my 2c are that this is worth investigating by owning sigs
i don't know what else could be the cause here. kubeadm is crd agnostic.
completely agree this needs resolution, I'm just confused since is a really similar setup and looks completely solid The test waits until the status is set, so even a resource-constrained env should just wait a little longer ... the failures I see are bailing out of that loop after only one wait (105ms), so something is turning that field non-nil without populating the expected results I'm also pretty confused
the diff between the two jobs (kinder/kind) reveals that the kind job has much higher resource limits/requests (for CPU) full diff: <details</details
i'm not familiar with this area of the code. it does appear to be waiting for the TypeChecking struct to be populated, but not for the Warnings slice to be populated. logically (without trying to find the code): if TypeChecking is expected to be populated with Warnings during a Get call after the Create call, we are catching a flake bug. if the Warnings in the TypeChecking are populated async of the TypeChecking then we are catching a test flake and the test must be updated to wait for the Warnings too.
well, the code is here: it appears that the only way for this to happen is if the Check() call returns nil warnings under "some conditions". the problem is not reproducible in the unit test here if the interval is set to a lower value <= 100ms and tested with the tool. but that's with a fake client.

this extra err check seems redundant: it might be worth temporary adding a print of the crd and policy before this line: then allow this test to flake again and inspect the results.
/milestone v1.30
Thanks for raising the issue and thanks for the investigation! I don't know exactly why the solid test in kind would be flaky in kubeadm by a quick scan but agree it needs attention.(The tests look less flaky though..) /assign the type checking code owner. Would you mind adding a printout as suggested above and see if we could get more info? Thanks!
note, both kind and kinder (from the compared jobs) use kubeadm.
and i wonder if this is an informer cache issue.
Same guess as failed for missing event for service patch since 03/06(similar time). [Not sure; just guess]
shows that the test failed for a different reason. failure in ci-kubernetes-e2e-ec2-alpha-enabled-default: from 03/04 to 03/06.
/remove-kind failing-test /kind flake
the API field doc indicates a non-nil value means type checking is complete: It doesn't look like the type checker has any way to signal that it was lacking some information needed to complete type checking, and will record an empty set of warnings if it is: are there any scenarios where a laggy REST mapper or informer can make the controller think it completed type checking but be missing some info that makes it produce an empty list of warnings? I think our options are: the test to wait for non-empty warnings in the status (not just non-nil typeChecking) the controller to not write an empty typeChecking stanza if it fails to type check in some way option 2 seems better for observers, but I don't know how invasive that is
option 2 feels like we are creating a hole? (folks will see warnings sometimes and sometimes they don't?)
Option 2 might cause confusion especially when deleting existing warnings from typechecking field.

(We will remove the typechecking warning if observed the fix in schema)
I'm not suggesting deleting existing warnings, but it appears we are currently reporting an empty typeChecking stanza when we have not actually finished type checking... that doesn't match the API docs and is likely confusing to API clients (it's confusing our own e2e test we wrote based on the documented behavior)
Ignoring errors while finishing the type checking is intended behavior defined by the original KEP. Can we go back to change the tests to do a retry loop until the CRD appears in the RESTMapper? The root problem of laggy RESTMapper seems out of the scope of this test case.
if is fixing the test then it's masking a state not guaranteed by the field comment: i guess already explained that above.
The type checking is completed with this field set to non-nil. The problem is that the type checking finishes with error, and it cause some of the referred types to be ignored.
should this error be surfaced to the user and does that need an e2e test?
No. The KEP defines that errors will be ignored. However, there is a next step to make the type checker reacts on CRD changes, which handles the case when the CRD schema is late for the type checking.
i think users might be confused by this behavior (the same way people on this thread are) even if it's documented in a KEP, so a field comment update feels worth it, at minimum.
The current type checking is informational and only works on ValidatingAdmissionPolicy. If the CRD is created after type checking or CRD created but not showing in schema watcher yet, the error would be ignored. And it will not retry as long as ValidatingAdmissionPolicy remains unchanged. There is a PR for adding retry with schema resolver raised in which will solve the issue entirely. In current case, the guess would be schema watcher is slow so it didn't have crd info yet when performing typechecking. The temporary fix by in will resolve the test flaky by adding manually retry. The field comment of makes sense to me since it indicates the completion of the type checking but not handling the case where the CRD has not been created or created but not showing in schema watcher yet. FYI The says I agree it might not fit in conformance test. Not sure about the conformance test criteria.
cc
/triage accepted


Question: return links, nil } <ins> // GetMountRefs : empty implementation here since there is no place to query all mount points on Windows </ins> func (mounter *Mounter) GetMountRefs(pathname string) ([]string, error) { if _, err := os.Stat(normalizeWindowsPath(pathname)); os.IsNotExist(err) { return []string{}, nil } else if err != nil { return nil, err } <del> refs, err := getAllParentLinks(normalizeWindowsPath(pathname)) if err != nil { return nil, err } return refs, nil </del> <ins> return []string{pathname}, nil </ins> } // Note that on windows, it always returns 0. We actually don't set FSGroup on

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: After creating pods having PersistentVolumeClaims and deleting them, the logs are getting filled with errors regarding failing to unmount the volume [1]. Additionally, tests requiring PVCs will hang several minutes while waiting the PV to unmount [2]. [1] [2] What you expected to happen: There shouldn't be any issues while unmounting the volume, the tests shouldn't have to wait several minutes waiting for the volume to be unmounted. How to reproduce it (as minimally and precisely as possible): pod requiring PVC. the pod. logs for error messages regarding the volume not being able to be unmounted. Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.2 Cloud provider or hardware configuration: Azure, acs-engine v0.21.2 (: ) OS (e.g. from /etc/os-release): Windows Server 1803 Kernel (e.g. ): Install tools: Others:
/sig azure /assign
what's your PVC type? is it azure disk?
thanks for reporting. I have identified that this is a regression introduced by UnmountDevice common code mainly for linux: the root cause is is not well implemented Error code is like following: I have a different implementation for func on windows wich could fix this issue, will file a PR tomorrow.


Question: return nil } <del> func convert_v1_StatusCause_To_api_StatusCause(in *StatusCause, out *api.StatusCause, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*StatusCause))(in) } out.Type = api.CauseType(in.Type) out.Message = in.Message out.Field = in.Field return nil } func convert_api_StatusCause_To_v1_StatusCause(in *api.StatusCause, out *StatusCause, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*api.StatusCause))(in) } out.Type = CauseType(in.Type) out.Message = in.Message out.Field = in.Field return nil } func convert_v1_StatusDetails_To_api_StatusDetails(in *StatusDetails, out *api.StatusDetails, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*StatusDetails))(in) } out.ID = in.ID out.Kind = in.Kind if in.Causes != nil { out.Causes = make([]api.StatusCause, len(in.Causes)) for i := range in.Causes { if err := convert_v1_StatusCause_To_api_StatusCause(&in.Causes[i], &out.Causes[i], s); err != nil { return err } } } else { out.Causes = nil } out.RetryAfterSeconds = in.RetryAfterSeconds return nil } func convert_api_StatusDetails_To_v1_StatusDetails(in *api.StatusDetails, out *StatusDetails, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*api.StatusDetails))(in) } out.ID = in.ID out.Kind = in.Kind if in.Causes != nil { out.Causes = make([]StatusCause, len(in.Causes)) for i := range in.Causes { if err := convert_api_StatusCause_To_v1_StatusCause(&in.Causes[i], &out.Causes[i], s); err != nil { return err } } } else { out.Causes = nil } out.RetryAfterSeconds = in.RetryAfterSeconds return nil } </del> func convert_v1_TCPSocketAction_To_api_TCPSocketAction(in *TCPSocketAction, out *api.TCPSocketAction, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*TCPSocketAction))(in)

When accessing a non existing entity (example - namespace), the message specifies the requested field as 'id', but it should be 'name'. Example (see the details section) { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "namespaces "staging" not found", "reason": "NotFound", "details": { "id": "staging", "kind": "namespaces" }, "code": 404 }
Yes. We should rename "id" to "name" in StatusDetails: Not sure if we can change v1beta3 API now. cc
You can add a new field and fill in both, and MAYBE remove ID in v1 , Nikhil Jindal wrote:
I suggest we fix it at most in v1 only. This is an annoying wart, but we could just fix the description string for now.
If trivial, we should fix v1.


Question: "STANDARD_M32LS":         32, "STANDARD_M32MS":         32, "STANDARD_M32TS":         32, <ins> "STANDARD_M416-208MS_V2": 64, "STANDARD_M416-208S_V2":  64, </ins> "STANDARD_M416MS_V2":     64, "STANDARD_M416S_V2":      64, "STANDARD_M64-16MS":      64,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: var err error wait.ExponentialBackoff(backoff, func() (bool, error) { err = webhookFn() if ctx.Err() != nil { // we timed out or were cancelled, we should not retry return true, err } <del> // these errors indicate a transient error that should be retried. if net.IsConnectionReset(err) || apierrors.IsInternalError(err) || apierrors.IsTimeout(err) || apierrors.IsTooManyRequests(err) { return false, nil } // if the error sends the Retry-After header, we respect it as an explicit confirmation we should retry. if _, shouldRetry := apierrors.SuggestsClientDelay(err); shouldRetry { </del> <ins> if shouldRetry(err) { </ins> return false, nil } if err != nil {

What would you like to be : The api server should retry sending audit events to configured webhook sinks regardless of the type of HTTP error code it might receive from the webhooks. Why is this needed: While some components may live with failed data transmissions, this is a particularly sensitive topic for the audit area. The API server audit mechanisms must ensure that no other reasonable option exists before dropping an audit event that may be the only trace of malicious behavior in the cluster. It seems that the current implementation may be improved further in this direction. Currently, when the API server is configured for non-blocking sending of audit events to a webhook audit sink backend, it will retry up to 5 times with exponential backoff, should there be an error in communicating with the webhook, as implemented . However, not all errors will trigger a retry. Some are considered transient and yield retry. Others simply stop the retry cycle. For example, it retries on receiving HTTP code , but not on receiving . The list of error codes that will yield a retry is actually quite small - three HTTP errors and one network error. Any other error will not trigger a retry and the audit events will be lost. The code comments explain that the retry will consider only recoverable failures, which may make sense in other cases, but as far as sending audit events to an audit backend webhook is concerned, it is not sufficient policy. A more robust approach would be to consider any error and give the backend service a chance to recover instead of relying on the error codes semantics and the webhook implementations interpretation of it. That is more inline with having the transmission of audit events as top priority. /sig auth /sig api-machinery


Question: "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//staging/src/k8s.io/component-base/config:go_default_library", ], )

What would you like to be : Actually disable scheduler plugins according to the disabled list in the scheduler's component config. The semantics are defined here: Part of /sig scheduling /priority important-soon /assign
The following semantics seems vague, and I think we should reach consensus on some cases when disabled plugins are involved. Suppose we have two default plugins, and , the runs before , here are the cases we could meet with the scheduling framework, plugins which run after all the default plugins plugins which should run ahead of the plugins plugin which should run ahead of all the plugins -enable default or custom plugin should return errors
Do you have any other cases not mentioned here?
Those cases sound right, I don't have others.
Is this actually what we want? I don't think so. The comment needs rewording.
I plan to revisit the comments after


Question: framework.ExpectNoError(err, "failed to wait for pod1 running status") l.pods = append(l.pods, pod1) <del> // Create the second pod, which will fail scheduling because the ReadWriteOncePod PVC is already in use. </del> <ins> // Create the second pod, which will preempt the first pod because it's using the // ReadWriteOncePod PVC and has higher priority. </ins> pod2, err := e2epod.MakeSecPod(&podConfig) framework.ExpectNoError(err, "failed to create spec for pod2") <ins> pod2.Spec.PriorityClassName = l.priorityClass.Name </ins> _, err = l.cs.CoreV1().Pods(pod2.Namespace).Create(ctx, pod2, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod2") <del> err = e2epod.WaitForPodNameUnschedulableInNamespace(ctx, l.cs, pod2.Name, pod2.Namespace) framework.ExpectNoError(err, "failed to wait for pod2 unschedulable status") </del> l.pods = append(l.pods, pod2) <del> // Delete the first pod and observe the second pod can now start. err = e2epod.DeletePodWithWait(ctx, l.cs, pod1) framework.ExpectNoError(err, "failed to delete pod1") </del> <ins> // Wait for the first pod to be preempted and the second pod to start. err = e2epod.WaitForPodNotFoundInNamespace(ctx, l.cs, pod1.Name, pod1.Namespace, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod1 to be preempted") </ins> err = e2epod.WaitTimeoutForPodRunningInNamespace(ctx, l.cs, pod2.Name, pod2.Namespace, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod2 running status") <ins> // Recreate the first pod, which will fail to schedule because the second pod // is using the ReadWriteOncePod PVC and has higher priority. _, err = l.cs.CoreV1().Pods(pod1.Namespace).Create(ctx, pod1, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod1") err = e2epod.WaitForPodNameUnschedulableInNamespace(ctx, l.cs, pod1.Name, pod1.Namespace) framework.ExpectNoError(err, "failed to wait for pod1 unschedulable status") // Delete the second pod with higher priority and observe the first pod can now start. err = e2epod.DeletePodWithWait(ctx, l.cs, pod2) framework.ExpectNoError(err, "failed to delete pod2") err = e2epod.WaitTimeoutForPodRunningInNamespace(ctx, l.cs, pod1.Name, pod1.Namespace, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod1 running status") </ins> }) ginkgo.It("should block a second pod from using an in-use ReadWriteOncePod volume on the same node", func(ctx context.Context) {

For 1.27 we should graduate the ReadWriteOncePod feature to beta. This is needed to rollout the feature to GA. In beta this feature will support preemption of pods using PVCs. See:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /sig scheduling


Question: source "${KUBE_ROOT}/test/cmd/storage.sh" source "${KUBE_ROOT}/test/cmd/template-output.sh" source "${KUBE_ROOT}/test/cmd/version.sh" <ins> source "${KUBE_ROOT}/test/cmd/wait.sh" </ins> ETCD_HOST=${ETCD_HOST:-127.0.0.1}

We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)


Question: }, }, { name:        "container_cpu_usage_seconds_total", <del> help:        "Cumulative cpu time consumed per cpu in seconds.", </del> <ins> help:        "Cumulative cpu time consumed in seconds.", </ins> valueType:   prometheus.CounterValue, extraLabels: []string{"cpu"}, getValues: func(s *info.ContainerStats) metricValues { <ins> if len(s.Cpu.Usage.PerCpu) == 0 { if s.Cpu.Usage.Total > 0 { return metricValues{{ value:  float64(s.Cpu.Usage.Total) / float64(time.Second), labels: []string{"total"}, }} } } </ins> values := make(metricValues, 0, len(s.Cpu.Usage.PerCpu)) for i, value := range s.Cpu.Usage.PerCpu { if value > 0 {

/kind feature This is the once-per-release cut of a new cAdvisor release, which corresponds to the next kubernetes release. cAdvisor v0.29 corresponds to kubernetes v1.10. /sig node /priority important-soon
self-reminder to add per-cpu stats to ignore metrics when vendoring. reminder to add a release notice that one of the container labels fields is being removed.
/kind bug turns out most of the changes are bug-fixes
please mark this issue as approved-for-milestone, since it certainly seems to be, and it's now code slush. Thanks!


Question: ginkgo.BeforeEach(func(ctx context.Context) { ginkgo.By("Wait for node to be ready") <del> gomega.Eventually(ctx, func(ctx context.Context) error { _, err := e2enode.TotalReady(ctx, f.ClientSet) if err != nil { return err } return nil }, time.Minute, time.Second).Should(gomega.Equal(1), "one node should be ready") </del> <ins> gomega.Eventually(ctx, e2enode.TotalReady). WithArguments(f.ClientSet). WithTimeout(time.Minute). Should(gomega.BeEquivalentTo(1)) </ins> // Before we run the device plugin test, we need to ensure // that the cluster is in a clean state and there are no

ci-crio-cgroupv1-node-e2e-flaky [It] [sig-node] Device Plugin [NodeFeature:DevicePlugin] [Serial] DevicePlugin [Serial] [Disruptive] Keeps device plugin assignments across node reboots (no pod restart, no device plugin re-registration) [Flaky] Since Under evaluation. cc /sig node
/triage accepted /priority important-soon
Tests are green again:


Question: "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", "//vendor/github.com/emicklei/go-restful:go_default_library", "//vendor/github.com/go-openapi/spec:go_default_library", <ins> "//vendor/k8s.io/kube-openapi/pkg/aggregator:go_default_library", </ins> "//vendor/k8s.io/kube-openapi/pkg/builder:go_default_library", "//vendor/k8s.io/kube-openapi/pkg/common:go_default_library", "//vendor/k8s.io/kube-openapi/pkg/util:go_default_library",

What happened: Created a CRD with a group/version/kind that defines a CustomResourceDefinition type overlapping with the in-tree kubernetes type (WARNING: do not create this CRD on a cluster you care about, see ) Note that two definitions for the group/version/kind appear in openapi: What you expected to happen: published openapi only includes the in-tree type, which has precedence /cc /area custom-resources /priority important-soon /sig api-machinery
/cc
haven't looked into the code yet. I'd expect the naming controller / apiextensions server to prevent conflicting CR GVK, but they are unaware of builtin API groups
CRD handling is isolated to the apiextensions server... it does not have information about all other built-in and aggregated APIs
mentioned something he was working on to make the aggregation layer honor APIService priorities (so that later definitions would not be included if something had already defined a GVK) edit: possibly
a naive idea is to reserve k8s groups in a hard code way that sounds good the CRD in defines a but the kubectl results are about . Did I miss something?
oops... fixed. I started with an API type under but the issue only surfaces for specific APIs (due to the generated definition name colliding for some types)
that wouldn't address duplicate GVKs from aggregated APIs
Three kinds of openapi paths/definitions that we want to prevent CRDs from overwriting: CRD API the spec aggregation in apiextensions-apiserver allows overwriting. I will send a fix native APIs well protected. The spec aggregation in kube-aggregator: 1) ignores duplicated paths and renames duplicated definitions; 2) treats kube-apiserver with than apiextensions-apiserver APIs the spec aggregation in kube-aggregator allows overwriting. The spec aggregation in kube-aggregator treats aggregated APIs with than apiextensions-apiserver. I'm thinking about giving apiextensions-apiserver the lowest priority, but then the problem is aggregated APIs can potentially overwrite the CRD API Note that none of the above prevents CRDs from adding new subpaths to existing APIs. E.g. a CRD can add a path to the CRD API, or to a aggregated API if it didn't have one


Question: t.Errorf("3: %v: diff: %vnCodec: %#v", name, diff.ObjectReflectDiff(object, obj3), codec) return } <ins> // do structure-preserving fuzzing of the deep-copied object. If it shares anything with the original, // the deep-copy was actually only a shallow copy. Then original and obj3 will be different after fuzzing. // NOTE: we use the encoding+decoding here as an alternative, guaranteed deep-copy to compare against. ValueFuzz(object) if !apiequality.Semantic.DeepEqual(original, obj3) { t.Errorf("0: %v: fuzzing a copy altered the original, diff: %v", name, diff.ObjectReflectDiff(original, object)) return } </ins> } // dataAsString returns the given byte array as a string; handles detecting

The apimachinery roundtripping test does not ensure that deep copies work. We need to provide a deep copy test that makes sure we don't end up with shallow copies.
I looked into gofuzz some days ago and noticed that it does not support value-only fuzzing that preserves object structure (= does not change pointer values). So we need something clever, maybe reflection based here. Any other idea?