Question: w.pod.Spec.RestartPolicy != v1.RestartPolicyNever } <ins> // Graceful shutdown of the pod. if w.pod.ObjectMeta.DeletionTimestamp != nil && (w.probeType == liveness || w.probeType == startup) { klog.V(3).Infof("Pod deletion requested, setting %v probe result to success: %v - %v", w.probeType.String(), format.Pod(w.pod), w.container.Name) if w.probeType == startup { klog.Warningf("Pod deletion requested before container has fully started: %v - %v", format.Pod(w.pod), w.container.Name) } // Set a last result to ensure quiet shutdown. w.resultsManager.Set(w.containerID, results.Success, w.pod) // Stop probing at this point. return false } </ins> // Probe disabled for InitialDelaySeconds. if int32(time.Since(c.State.Running.StartedAt.Time).Seconds()) < w.spec.InitialDelaySeconds { return true

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: Addon objects are created in the "kube-system" namespace. <del> Example addons are: * [DNS](http://releases.k8s.io/HEAD/cluster/addons/dns/) provides cluster local DNS. * [kube-ui](http://releases.k8s.io/HEAD/cluster/addons/kube-ui/) provides a graphical UI for the cluster. * [fluentd-elasticsearch](http://releases.k8s.io/HEAD/cluster/addons/fluentd-elasticsearch/) provides log storage. Also see the [gcp version](http://releases.k8s.io/HEAD/cluster/addons/fluentd-gcp/). * [cluster-monitoring](http://releases.k8s.io/HEAD/cluster/addons/cluster-monitoring/) provides monitoring for the cluster. </del> <ins> #### DNS While the other addons are not strictly required, all Kubernetes clusters should have [cluster DNS](dns.md), as many examples rely on it. Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services. Containers started by Kubernetes automatically include this DNS server in their DNS searches. #### User interface The kube-ui provides a read-only overview of the cluster state.  Access [the UI using kubectl proxy](../user-guide/connecting-to-applications-proxy.md#connecting-to-the-kube-ui-service-from-your-local-workstation) #### Container Resource Monitoring [Container Resource Monitoring](../user-guide/monitoring.md) records generic time-series metrics about containers in a central database, and provides a UI for browsing that data. #### Cluster-level Logging [Container Logging](../user-guide/monitoring.md) saves container logs to a central log store with search/browsing interface.  There are two implementations: * [Cluster-level logging to Google Cloud Logging]( docs/user-guide/logging.md#cluster-level-logging-to-google-cloud-logging) * [Cluster-level Logging with Elasticsearch and Kibana]( docs/user-guide/logging.md#cluster-level-logging-with-elasticsearch-and-kibana) </ins> ## Node components

has TODOs for how to bring up cluster addons like monitoring and logging. We should fill these in. requested by user:


Question: ) // TODO: use restful.CrossOriginResourceSharing <del> // See github.com/emicklei/go-restful/blob/master/examples/restful-CORS-filter.go, and // github.com/emicklei/go-restful/blob/master/examples/restful-basic-authentication.go </del> <ins> // See github.com/emicklei/go-restful/blob/master/examples/cors/restful-CORS-filter.go, and // github.com/emicklei/go-restful/blob/master/examples/basicauth/restful-basic-authentication.go </ins> // Or, for a more detailed implementation use https://github.com/martini-contrib/cors // or implement CORS at your proxy layer.

What would you like to be : // See , and // right address of restful-CORS- and restful-basic- in the comment of Why is this needed: the address of restful-CORS- and restful-basic- in the comment of is wrong now, this may affect reading codes. Thanks.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /sig docs /assign


Question: // ServiceAnnotationAllowedServiceTag is the annotation used on the service // to specify a list of allowed service tags separated by comma <ins> // Refer https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#service-tags for all supported service tags. </ins> ServiceAnnotationAllowedServiceTag = "service.beta.kubernetes.io/azure-allowed-service-tags" // ServiceAnnotationLoadBalancerIdleTimeout is the annotation used on the service

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider has a limited support for service tags by annotations: But according to , there are actually much more service tags than this list. Hence, errors would happen if the service tag is not included in the above list. What you expected to happen: Azure cloud provider should support all service tags from Azure. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig azure /priority important-soon
/assign


Question: // Status.Replicas should go up from 2->4 even though we created 5-4=1 pod rc.Status = api.ReplicationControllerStatus{Replicas: 4} <ins> // These are set by default. rc.Spec.Selector = rc.Spec.Template.Labels rc.Labels = rc.Spec.Template.Labels </ins> decRc := runtime.EncodeOrDie(testapi.Codec(), &rc) fakeUpdateHandler.ValidateRequest(t, testapi.ResourcePathWithQueryParams(replicationControllerResourceName(), rc.Namespace, rc.Name), "PUT", &decRc) validateSyncReplication(t, &fakePodControl, 1, 0)

It doesn't make sense to define pod template with a set of labels that is strict subset of replicaSelector (or at least I can't find an use-case). For that reason I think that the labels in pod template part of replication controller should be optional and user should care about them only if he wants to define a superset of what replicaSelector already defines.
I've heard the same comments from a client. I think that it'd be more intuitive if we always specify pod labels in the template (so that we can copy&paste pod definition), and replicaSelector is optional (by default pod labels).
It would be reasonable for the selector to be a subset. I'm fine with intelligent defaulting but we need to be careful not to conflate the 2 concepts, which I think would increase the amount of confusion.
Setting RC selector from pod template labels SGTM. Any concerns It wouldn't have to work with separate templates, though it would be desirable if it did. It would be nice if we could add preconditions based on other objects (resource version of pod template in this case) to etcd transactions.
I think we agreed in another issue that defaulting the selector from the pod template labels is reasonable. No argument from me. One problem with pod templates is that we lose the strict relationship between selector and template, so if you change the template the RC needs to behave intelligently (I refuse to create more pods because the template selector doesn't match mine, etc)
We should do this, and we should default replicas to 1. We could also consider defaulting replication controller labels from pod template labels. That way, essentially the only thing that would need to be specified for a singleton RC would be the pod template.
I don't think we can default replicas to 1, since we can't tell the difference between an explicit zero and a default zero, and I think there are lots of reasonable cases where you want to set replicas == 0 on create. (esp. in the context of socket activation)
Responded on the PR: We can distinguish default from 0 if we make it a pointer.


Question: var diskControllerType = vs.cfg.Disk.SCSIControllerType // find SCSI controller of particular type from VM devices <del> var diskController = getSCSIController(vmDevices, diskControllerType) </del> <ins> allSCSIControllers := getSCSIControllers(vmDevices) scsiControllersOfRequiredType := getSCSIControllersOfType(vmDevices, diskControllerType) scsiController := getAvailableSCSIController(scsiControllersOfRequiredType) </ins> var newSCSICreated = false var newSCSIController types.BaseVirtualDevice // creating a scsi controller as there is none found of controller type defined <del> if diskController == nil { </del> <ins> if scsiController == nil { if len(allSCSIControllers) >= SCSIControllerLimit { // we reached the maximum number of controllers we can attach return "", "", fmt.Errorf("SCSI Controller Limit of %d has been reached, cannot create another SCSI controller", SCSIControllerLimit) } </ins> glog.V(4).Infof("Creating a SCSI controller of %v type", diskControllerType) newSCSIController, err := vmDevices.CreateSCSIController(diskControllerType) if err != nil {

vSphere cloud provider volume plugin attach disk limits the number of volumes because of two issues: new SCSI controller is only created when one does not exist. A new controller should be when required and limit not reached. disk will fail when attaching disk to slot 7 on controller. Slot 7 on SCSI controller is reserved and therefore should be skipped. This issue has been brought up in Govmomi library .


Question: } return destroyTask.Wait(ctx) } <ins> // Remove the cluster or folder path from the vDiskPath // for vDiskPath [DatastoreCluster/sharedVmfs-0] kubevols/e2e-vmdk-1234.vmdk, return value is [sharedVmfs-0] kubevols/e2e-vmdk-1234.vmdk // for vDiskPath [sharedVmfs-0] kubevols/e2e-vmdk-1234.vmdk, return value remains same [sharedVmfs-0] kubevols/e2e-vmdk-1234.vmdk func removeClusterFromVDiskPath(vDiskPath string) string { datastore := regexp.MustCompile("[(.*?)]").FindStringSubmatch(vDiskPath)[1] if filepath.Base(datastore) != datastore { vDiskPath = strings.Replace(vDiskPath, datastore, filepath.Base(datastore), 1) } return vDiskPath } </ins>

With version 1.4.7 I could use a datastore that is in a cluster as a vsphere volume as . When I upgraded to 1.6.0, this same exact path does not work and throws a error. I'm struggling to find what changed, but am guessing it's difference in how the datastore is being looked up in vcenter. Has anyone had luck with this?
We have verified creating pod using static PV and dynamic provisioning, both works fine with Can you post here? cc:
Did that include using a datastore that was in a datastore cluster? That seemed to be working before because the datastore previously (before 1.5) was read out of the global config, which supported datastore path with cluster name. But post 1.5 datastore is parsed from the entry, which if you include the path to the cluster it gets you past the datastore lookup, but then when you go to add the disk it fails because that is not valid to vCenter. It's kind of this weird mixed mode, because to look up the datastore you need the path including the cluster, but to create the file you need the path without the cluster. I presume the PV is working because they use the cloud config values? I was not using a PV but putting a volumePath directly in my deployment yaml. Should I be using a PV to do this? If the deployment is deleted and re-created how do I guarantee that my pod would get the same PV claim? That was why I did not go down that path originally. I'm pretty sure I have narrowed down where in the provider this is happening, but didn't see a good path to get it to work. I have, admittantly, not tried PV/PVCs yet, due to the concern raised above, but could try those to see. The pertinent part of the deployment yaml: The cloud config:
Just confirmed I have the same issue using a PV and PVC....
Here it needs it in format: (which it gets on line 791 by parsing . But here it needs it in true format, without the datastorecluster: Therefore if you do it gets you past line 796 but fails on 802. But if you do then you don't get past line 796.
Thanks for detailed look up. This is the commit which caused this regression. We will work on the fix.

cc:
Ah yes, I see that now. What is the lifecycle of a dynamic disk? Does that disk live on after the PVC is released? After the PV is deleted? It seems like I may just not fully be exercising the power of the volume plugins as well. I would still have the issue with existing volumes as they have data that I need to maintain, but would be interesting for new volumes to look into these more.
RECLAIMPOLICY for PV created using Dynamic Provisioning is Delete PV will get deleted if you delete PVC. Disk also gets deleted from datastore.


Question: if len(src.File) == 0 || len(dest.File) == 0 { return errFileCannotBeEmpty } <ins> if _, err := os.Stat(src.File); err != nil { return fmt.Errorf("%s doesn't exist in local filesystem", src.File) } </ins> reader, writer := io.Pipe() // strip trailing slash (if any)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running silently fails if that source file doesn't exist. What you expected to happen: I expected kubectl to exit with a non-zero status code and to print a standard error message like so: How to reproduce it (as minimally and precisely as possible): Find any running pod, try to copy a non-existent file: Anything else we need to know?: works right. For example: I'm sure this has something to do with the fact that uses under the hood. However, itself will report an error if you try to extract a file that doesn't exist, so maybe this is just a matter of not including the to-be-extracted-path(s) on the end of the tar command line. This might be fixed by appending (or rather the basename of it) to the end of the array here: (haven't tried it). Environment: Kubernetes version (use ): and Cloud provider or hardware configuration: docker-for-desktop 2.0.4.1 () and GKE running 1.13.6-gke-0 OS (e.g: ): macOS 10.14.5 Kernel (e.g. ): Install tools: Installed docker from dmg, using the included Network plugin and version (if this is a network-related bug): Others:
/assign /sig cli
any update on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Same issue when the file doesn't exist on the pod
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Beep
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
This is the first example where I found the kubectl cli to be sub-par (otherwise the experience has been excellent). Is anyone looking at this?
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For me never copies anything and never writes any info to console. Even if I point it to non-existing pod, it just exists with no trace.
I've opened with a fix.
: what version of kubectl are you using? I tried to upload a file to a non-existent pod and it failed with an error message (using the master branch) :
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Any maintainer here? I had opened with a fix.
Would be great if would return a simple error in the CLI. Our pipeline was failing for a month and none realised because all the kubectl results were green :heavycheckmark:


Question: // create custom endpoint slices tcpProtocol := v1.ProtocolTCP <ins> readyCondTrue := true </ins> epsTemplate := &discoveryv1.EndpointSlice{ ObjectMeta: metav1.ObjectMeta{GenerateName: "e2e-custom-slice", Labels: map[string]string{

The test create custome EndpointSlices to validate some behaviors sending traffic, so it is clear that it expects the EndpointSlices to be ready, The API doc is vague about this situation and leaves room to interpret the "unknown behavior", quoting Since the test expects the EndpointSlices to be Ready we should set them, if we decide to change the API behavior, then we should add an additional e2e test to validate this behavior, but not use the same test to validate two different things /kind bug /kind failing-testing /sig network Slack thread
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : The test create custome EndpointSlices to validate some behaviors sending traffic, so it is clear that it expects the EndpointSlices to be ready, /kind failing-testing /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: aauren. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks I'll take this one and draft up a PR this weekend. (this comment should hopefully allow the k8s ci robot to assign this to me as well)
if you are planning to contribute more often you should apply as org member


Question: }, }, }, <ins> "catch all with path of type prefix and secret name": { rules: []string{ "/path*=catchall:8080,tls=secret1", }, ingressclass:   ingressClass, defaultbackend: "service1:https", annotations:    []string{}, expected: &networkingv1.Ingress{ TypeMeta: metav1.TypeMeta{ APIVersion: networkingv1.SchemeGroupVersion.String(), Kind:       "Ingress", }, ObjectMeta: metav1.ObjectMeta{ Name:        ingressName, Annotations: map[string]string{}, }, Spec: networkingv1.IngressSpec{ IngressClassName: &ingressClass, DefaultBackend: &networkingv1.IngressBackend{ Service: &networkingv1.IngressServiceBackend{ Name: "service1", Port: networkingv1.ServiceBackendPort{ Name: "https", }, }, }, TLS: []v1.IngressTLS{ { SecretName: "secret1", }, }, Rules: []networkingv1.IngressRule{ { Host: "", IngressRuleValue: networkingv1.IngressRuleValue{ HTTP: &networkingv1.HTTPIngressRuleValue{ Paths: []networkingv1.HTTPIngressPath{ { Path:     "/path", PathType: &pathTypePrefix, Backend: networkingv1.IngressBackend{ Service: &networkingv1.IngressServiceBackend{ Name: "catchall", Port: networkingv1.ServiceBackendPort{ Number: 8080, }, }, }, }, }, }, }, }, }, }, }, }, </ins> "mixed hosts with mixed TLS configuration and a default backend": { rules: []string{ "foo.com/=foo-svc:8080,tls=",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl create ingress panics because there is no cmdutil.AddDryRunFlag What you expected to happen: Command to work How to reproduce it (as minimally and precisely as possible): Compile the upstream, run ` Anything else we need to know?: Environment: Kubernetes version (use ): upstream/dev /sig cli /kind bug /priority critical-urgent
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "--proxy-client-key-file=/var/run/kubernetes/proxy.key", "--request-timeout=2m", "--storage-backend=etcd3", <ins> "--service-cluster-ip-range=192.168.128.0/17", </ins> } fs.Parse(args) // This is a snapshot of expected options parsed by args. expected := &ServerRunOptions{ ServiceNodePortRange:   kubeoptions.DefaultServiceNodePortRange, <del> ServiceClusterIPRanges: kubeoptions.DefaultServiceIPCIDR.String(), </del> <ins> ServiceClusterIPRanges: (&net.IPNet{IP: net.ParseIP("192.168.128.0"), Mask: net.CIDRMask(17, 32)}).String(), </ins> MasterCount:            5, EndpointReconcilerType: string(reconcilers.LeaseEndpointReconcilerType), AllowPrivileged:        false,

API server defaults the service CIDR. We should instead just fail if it is not specified. Defaulting is a wild guess and very likely to be wrong. /sig network /kind bug
For existing clusters that don't currently set it, is setting it to anything other than the current default a breaking change?
Yes. You can not really change it (well, you can expand it, but not totally move it), unless you delete ALL services, including kubernetes.default. I filed this aspirationally. It's not on fire, it's just a dumb thing to have provided a default for.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expectedErr bool }{ { <del> desc:        "an error shall be returned if there's no such instance corresponding to given nodeName", </del> <ins> desc:        "error should not be returned if there's no such instance corresponding to given nodeName", </ins> nodeName:    "vm1", <del> expectedErr: true, </del> <ins> expectedErr: false, </ins> }, { desc:        "no error shall be returned if there's no matching disk according to given diskName",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If a node with a Azure Disk attached is deleted (before the volume is detached), subsequent attempts by the attach/detach controller to detach it continuously fail, and prevent the controller from attaching the volume to another node. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Create a pod referencing a Azure Disk Wait for pod to get scheduled and running. Delete the node VM that the pod is scheduled to. Check if volume is detached by attach/detach controller: Expected: Volume detached. Actual: Volume continuously fails detach. Workaround: Restart the kube-controller-manager binary on the master. Error logs: Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure cc


Question: "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructuredscheme" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/json"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : add --list option to kubectl annotate command Why is this needed: like kubectl label commandshow annotations quickly
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign


Question: ) // ScalesGetter can produce a ScaleInterface <del> // for a particular namespace. </del> type ScalesGetter interface { <ins> // Scales produces a ScaleInterface for a particular namespace. // Set namespace to the empty string for non-namespaced resources. </ins> Scales(namespace string) ScaleInterface }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scale client () currently works fine with namespaced resources. However, it doesn't work for non-namespaced resources e.g. cluster-scoped CRDs. The current implementation should be extended in order to support non-namespaced resources. The current interface: /area custom-resources /sig autoscaling
I think scale subresource support for CRDs falls more into sig-apimachinery domain. After DirectXMan12 moved on from sig-autoscaling I don't think there is anyone in our sig-autoscaling with knowledge of scale subresource internals. /sig api-machinery
/assign
The bug report sounds legit. Does the generic scale client need to be changed (I think so), or does solve your issue?
No, the dashboard PR doesn't solve this. I discovered this issue while working on that PR so when gets merged, it will only work for namespaced CRDs. I think the generic scale client needs to be updated, maybe by adding another getter to the ScalesGetter interface: /kind bug
PR:


Question: map[string]string{ "type": "pd-standard", }, <del> "1.5Gi", "2Gi", </del> <ins> "1.5G", "2G", </ins> func(volume *v1.PersistentVolume) error { return checkGCEPD(volume, "pd-standard") },

GCE PDs are allocated in chunks of GBs (not GiB) but function incorrectly creates volume in chunks of GiB - cc /sig storage
/kind bug
/assign
/priority important-soon
Hey I submit a pr for this issue. Your suggestions are very important. Thanks.


Question: apiVersion: extensions/v1beta1 kind: Deployment metadata: <del> name: heapster-v1.5.0-beta.0 </del> <ins> name: heapster-v1.5.0-beta.1 </ins> namespace: kube-system labels: k8s-app: heapster kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile <del> version: v1.5.0-beta.0 </del> <ins> version: v1.5.0-beta.1 </ins> spec: replicas: 1 selector: matchLabels: k8s-app: heapster <del> version: v1.5.0-beta.0 </del> <ins> version: v1.5.0-beta.1 </ins> template: metadata: labels: k8s-app: heapster <del> version: v1.5.0-beta.0 </del> <ins> version: v1.5.0-beta.1 </ins> annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: containers: <del> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0 </del> <ins> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.1 </ins> name: heapster livenessProbe: httpGet:

I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue?

This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.


Question: } return false, nil }) <ins> if err != nil { e2essh.LogResult(result) } </ins> framework.ExpectNoError(err, "kubelet did not recreate its iptables rules") }) })

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: Networking should recreate its iptables rules if they are deleted [Disruptive] Since when has it been failing: Commit Testgrid link: Reason for failure: Anything else we need to know: Possibly related to this? It's listed in the git diff /sig network
/assign /cc Thanks for detecting it so quicklly
most probably is related to
/unassign /assign he has already submitted a fix
/triage accepted
Reviewed. Marking as resolved on the CI Signal Board. Once was merged, failures stopped and they have not re-occurred in the last 8 days (Oct 10th)
Yay


Question: ReadonlyPaths: securitycontext.ConvertToRuntimeReadonlyPaths(effectiveSc.ProcMount), } } <del> // TODO: Deprecated, remove after we switch to Seccomp field // set SeccompProfilePath. </del> var err error <del> synthesized.SeccompProfilePath, err = m.getSeccompProfilePath(pod.Annotations, container.Name, pod.Spec.SecurityContext, container.SecurityContext, m.seccompDefault) if err != nil { return nil, err } </del> synthesized.Seccomp, err = m.getSeccompProfile(pod.Annotations, container.Name, pod.Spec.SecurityContext, container.SecurityContext, m.seccompDefault) if err != nil {

Look for "// TODO: Deprecated, remove after we switch to Seccomp field". Fix it. See as example. /cc /help /good-first-issue /sig node add as a reviewer when sending a PR. (issue is a follow up from )
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Look for "// TODO: Deprecated, remove after we switch to Seccomp field". Fix it. See as example. /good-first-issue /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will start to look at it and fix part of them /assign
/assign
/assign
/assign
/assign
I would like to work on this issue. Can you assign this to me? /assign
/assign
Should I work on this Issue? /assign
/assign
Should I work on this Issue? /assign
I believe there is someone working on this now, and in approving phase, so it's kinda done.
Hey, I would like to work on this. /assign
i got it!! /assign
/assign


Question: package validation import ( <ins> "math" </ins> "net" "regexp" "strings"

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: framework.ExpectNoError(f.WaitForPodRunningSlow(host1Pod.Name)) <del> v, err := f.ReadFileViaContainer(host1Pod.Name, containerName, testFile) framework.ExpectNoError(err) framework.Logf("Read value: %v", v) Expect(strings.TrimSpace(v)).To(Equal(strings.TrimSpace(testFileContents))) </del> <ins> verifyPDContentsViaContainer(f, host1Pod.Name, containerName, map[string]string{testFile: testFileContents}) </ins> // Verify that disk is removed from node 1's VolumeInUse list framework.ExpectNoError(waitForPDInVolumesInUse(nodeClient, diskName, host0Name, nodeStatusTimeout, false /* shouldExist */))

https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gke-slow kubernetes-e2e-gke-test Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gke-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
Removing attention-needed label. This is currently under investigation. Nothing here appears to be release blocking, but we will continue to reassess as we investigate.
Builds: kubernetes-e2e-gce-slow kubernetes-e2e-gke-slow kubernetes-soak-continuous-e2e-gke Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gci-gce-slow kubernetes-e2e-gci-gke-slow-release-1.4 kubernetes-e2e-gci-gke-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
any news wrt these failures?
Yes, investigated and discovered () which appears to indicate that these flakes are caused by . Tracking issue in
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
Great, thanks!
Builds: kubernetes-e2e-gci-gce-slow-release-1.4 kubernetes-e2e-gci-gke-prod kubernetes-e2e-gci-gke-slow-release-1.4 Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

Builds: kubernetes-e2e-gci-gce-slow-release-1.4 kubernetes-e2e-gci-gce-slow kubernetes-e2e-gci-gke-slow-release-1.4 kubernetes-e2e-gci-gke-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
do you believe this is essentially a dup of , should this issue be closed?
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
They are different tests, but same root cause and solution. Let's leave them open until they are actually fixed.

Builds: kubernetes-e2e-gci-gce-slow-release-1.4 kubernetes-e2e-gci-gce-slow kubernetes-e2e-gci-gke-slow-release-1.4 kubernetes-e2e-gci-gke-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gci-gce-slow-release-1.4 kubernetes-e2e-gci-gke-slow-release-1.4 Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This is a non-blocker for 1.5 This test calls directly to read value instead of through which contains . We should maybe move the retry logic from to ? cc who may want to fix this.
Builds: ci-kubernetes-e2e-gci-gce-slow-release-1.4 ci-kubernetes-e2e-gci-gke-slow-release-1.4 ci-kubernetes-e2e-gci-gke-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: ci-kubernetes-e2e-gci-gke-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

"VolumesInUse Status to NOT contain diskName" is a new failure mode we haven't seen in flakes before. can you see if there was a recent change that might've triggered that?
Investigated with Saad the failures "VolumesInUse Status to NOT contain diskName", looks like this happened during the PR was merged. Due to an error caused by this PR, mounter script will fail to garbage collect the rocket container. UnmountDevice will fail because the global mount has other mount references. After is reverted, this error should not happen again.
Builds: ci-kubernetes-e2e-gci-gce-slow-release-1.4 ci-kubernetes-e2e-gci-gce-slow ci-kubernetes-e2e-gci-gke-slow-release-1.4 ci-kubernetes-e2e-gci-gke-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: ci-kubernetes-e2e-gci-gce-slow-release-1.4 ci-kubernetes-e2e-gci-gce-slow Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-slow ci-kubernetes-e2e-gke-containervm-1.3-gci-1.5-upgrade-cluster-new Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone?

(and remove the non-release-blocker tag as well)
https://k8s- Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
Yep
Builds: ci-kubernetes-e2e-gci-gce-slow-release-1.4 ci-kubernetes-e2e-gci-gce-slow-release-1.5 ci-kubernetes-e2e-gci-gce-slow ci-kubernetes-e2e-gci-gke-slow-release-1.4 Failed: [] Pod Disks Should schedule a pod w/ a RW PD, gracefully remove it, then schedule it on another host [Slow] {Kubernetes e2e suite}
CC
I'm working on a fix for this. It will change all the callers of kubectl exec in to use the retry wrapper instead.


Question: } func (f *fakeVolumeHost) WaitForKubeletErrNil() error { <del> return wait.PollImmediate(100*time.Millisecond, 10*time.Second, func() (bool, error) { </del> <ins> return wait.PollImmediate(10*time.Millisecond, 10*time.Second, func() (bool, error) { </ins> f.mux.Lock() defer f.mux.Unlock() return f.kubeletErr == nil, nil

What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary


Question: return controller } <del> // newTestPods creates a list of pods (specification) for test. func newTestPods(numPods int, volume bool, imageName, podType string) []*v1.Pod { </del> <ins> // newDensityTestPods creates a list of pods (specification) for test. func newDensityTestPods(numPods int, volume bool, imageName, podType string) []*v1.Pod { </ins> var pods []*v1.Pod for i := 0; i < numPods; i++ { podName := "test-" + string(uuid.NewUUID()) <del> labels := map[string]string{ "type": podType, "name": podName, </del> <ins> pod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, Labels: map[string]string{ "type": podType, "name": podName, }, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, }, </ins> } if volume { <del> pods = append(pods, &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   podName, Labels: labels, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, VolumeMounts: []v1.VolumeMount{ {MountPath: "/test-volume-mnt", Name: podName + "-volume"}, }, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, Volumes: []v1.Volume{ {Name: podName + "-volume", VolumeSource: v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}}, }, }, }) } else { pods = append(pods, &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   podName, Labels: labels, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, }, }) </del> <ins> pod.Spec.Containers[0].VolumeMounts = []v1.VolumeMount{ {MountPath: "/test-volume-mnt", Name: podName + "-volume"}, } pod.Spec.Volumes = []v1.Volume{ {Name: podName + "-volume", VolumeSource: v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}}, } </ins> } <ins> pods = append(pods, &pod) </ins> } return pods }

Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly.

If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the .

I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information?

The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation


Question: <ins> # Developing Kubernetes Docs in this directory relate to developing Kubernetes. * **On Collaborative Development** ([collab.md](collab.md)): info on pull requests and code reviews. * **Development Guide** ([development.md](development.md)): Setting up your environment; tests. * **Hunting flaky tests** ([flaky-tests.md](flaky-tests.md)): We have a goal of 99.9% flake free tests. Here's how to run your tests many times. * **GitHub Issues** ([issues.md](issues.md)): How incoming issues are reviewed and prioritized. * **Logging Conventions** ([logging.md](logging.md)]: Glog levels. * **Pull Request Process** ([pull-requests.md](pull-requests.md)): When and why pull requests are closed. * **Releasing Kubernetes** ([releasing.md](releasing.md)): How to create a Kubernetes release (as in version) and how the version information gets embedded into the built binaries. </ins>

master-blocking build-master ci-kubernetes-build.Overall ! 2023-02-07 22:06:39 +0000 UTC https://k8s- Link to failing run: make: [Makefile:374: release] Error 1 level=fatal msg="failed to run: running make release: command /usr/bin/make release did not succeed: line 604: ifconfig: command not foundn!!! [0207 23:28:23] Call tree:n!!! [0207 23:28:23] 1: hack/make- kube::golang::buildbinaries(...)nmake[1]: [Makefile:92: all] Error 1nmake: [Makefile:487: cross] Error 1n!!! [0207 23:28:25] Call tree:n!!! [0207 23:28:25] 1: kube::build::runbuildcommandex(...)n!!! [0207 23:28:25] 2: kube::build::runbuildcommand(...)nmake: [Makefile:374: release] Error 1n" Looks like the binary is not present in the test environment. No response /sig testing cc
/priority critical-urgent
/sig release /remove-sig testing looks like needs patching
/cc
I'm about to go offline for the night, probably just needs to add iproute2 package to the apt-get install. I would guess this is because during go1.20 upgrade it also went from buster to bullseye (Didn't confirm, but same issue going from buster to bullseye elsewhere, seems likely go1.20 included a newer debian base)
Thanks Ben!
/assign /triage accepted
/priority critical-urgent
contains only the utility. package has . Long term, we should investigate if we can eliminate using entirely as is touted as its replacement.
you can , do you have a link to the code so I can see if I can PR or suggest how to fix it?
/reopen till the current build passes
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
-- references to are across the repositories: The reference to in current context is in There's a check in which should switch to if that is present.
yeah, so that should work if doesn't exist
Yes, that's because even isn't installed.
The build failed again. Investigating:
On a first pass, this looks similar to
I tried building/cross-compiling the server test binaries (which is where the build errored) on an Ubuntu linux/amd64 machine. This seems to be working. Wondering if there's a build environment specific issue.
I tried on latest debian with a cross compile toolchain and golang 1.20 and was successful as well. Do we want to try some workaround like:
I am also thinking to do that. Digging into our toolchain right now.
PTAL at
that looks good. let's try that.
It's green now!
Hm, looks like magically fixed the relocation problem. Ugh, that's odd and it feels that the issue can occur depending on how we change the source code. :roll_eyes: I can still reproduce the issue with , so the test results are correct. !
yeah, that reminds me of where seemingly meaningless code changes fixed a compiler issue do we have a smaller reproducer to give to help with investigation?
Unfortunately not, but I a patch to focus on the failing test build target.
hoisting from do we have the full expanded invocation (including all the goflags / gogcflags / goasmflags / goldflags / tags) that was breaking?

that would help in giving the go team a reproducer
I'm unable to reproduce locally with the following are we sure this is a go issue, or is it an issue with the ARM cross-compiler (), since the cross-compile env changed versions around the same time as the go bump according to
Inside of the kube-build image it's reproducible with: It's reproducible with and
Reproducer with a Dockerfile, on k/k : It breaks with go 1.20 but works with go 1.19.5 on debian:latest (bullseye-, arm-linux-gnueabihf-gcc 10.2.1) It breaks with go 1.20 but works with go 1.19.5 on debian:testing (testing-, arm-linux-gnueabihf-gcc 12.2.0)
xref Seems to be happening again:


Question: "github.com/pkg/errors" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/validation" certutil "k8s.io/client-go/util/cert" "k8s.io/client-go/util/keyutil"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Duplicate altnames in certs which generated by kubeadm ! What you expected to happen: No Duplicate altnames in certs which generated by kubeadm How to reproduce it (as minimally and precisely as possible): Anytime Anything else we need to know?: ! Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
/area kubeadm
Can you describe your detailed usage scenarios and reproduction steps, and explain the kubernetes version so that we can quickly locate
Thank for your response! I have found why and will fix it soon.
i have commented on the PR. /priority awaiting-more-evidence
/remove-priority awaiting-more-evidence /priority backlog


Question: srcs = [ "arguments_test.go", "cgroupdriver_test.go", <del> "chroot_test.go", </del> "endpoint_test.go", "error_test.go", "marshal_test.go",

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: () Since when has it been failing: Failing 4/27. Testgrid link: Reason for failure: /kind failing-test /priority critical-urgent /milestone v1.15 /sig cluster-lifecycle /cc
cc
testing locally
PR sent: feel free to LGTM.
Thank you very much and for jumping on this so quickly


Question: v := version.Get() headers.Set("User-Agent", fmt.Sprintf("kube-probe/%s.%s", v.Major, v.Minor)) } <ins> if _, ok := headers["Accept"]; !ok { // Accept header was not defined. accept all headers.Set("Accept", "*/*") } if headers.Get("Accept") == "" { // Accept header was overridden but is empty. removing headers.Del("Accept") } </ins> req.Header = headers if headers.Get("Host") != "" { req.Host = headers.Get("Host")

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I would like that besides User-agent we the Accept header as well with the "acept all" default. Just like curl does. Why is this needed: Because it was my expectation that the probe would work similarly to a curl but is not the case. I think more folks can hit this situation and waste some time to finally sort it out If this is not something you would like to add I would then like the docs to mention this specifically. _ PS: I would love to propose the improvement my self
/sig node network
/assign
Is the goal simply to avoid tripping over servers that look for that header? I don't think this presents any hazards - is that right? If so, it seems like a simple PR if you want to tackle it. /triage accepted /kind feature
cool! will look into it! thanks :)
Hello sorry but be bugging you, but I have opened the PR but I had 0 response since then. I think I have met all the requirements, so I was wondering (as this is my first PR in k8s) if you could help me a bit. Thanks anyway.


Question: if s.Empty() { return r } <del> return r.setParam(api.FieldSelectorQueryParam(r.apiVersion), s.String()) </del> <ins> s2, err := s.Transform(func(field, value string) (newField, newValue string, err error) { return fieldMappings.filterField(r.apiVersion, r.resource, field, value) }) if err != nil { r.err = err return r } return r.setParam(api.FieldSelectorQueryParam(r.apiVersion), s2.String()) </ins> } // LabelsSelectorParam adds the given selector as a query parameter

My broke it for v1beta3, then fixed it for v1beta3 and broke it for v1beta1. I'm working on a fix.
Honestly I'm very surprised that more people aren't complaining-- integration test logs are full of noise because of this.
Exact text of error log is "Failed to watch *api.Node: field label not supported: "
cc


Question: unix.Close(n.epfd) close(n.stop) } <ins> // disabledThresholdNotifier is a fake diasbled threshold notifier that performs no-ops. type disabledThresholdNotifier struct{} func (*disabledThresholdNotifier) Start(_ chan<- struct{}) {} func (*disabledThresholdNotifier) Stop()                   {} </ins>

Start kubelet on a cgroupv2 image with kubelet config option or the flag enabled. Kubelet will periodically log something like: This is because the kernel memcg notifications relies on kubelet's which uses evenfd on file which is cgroupv1 specific and not supported on cgroupv2. Kubelet will disable kernel memcg notifications on cgroupv2 if they are unsupported and consider long term support for more cgroupv2 "compatible" ways to monitor for memory pressure. The closest analogous cgroupv2 concept is using inotify notifications on file and or combination with PSI pressure. This requires to be set on pod cgroup which requires the MemoryQOS feature to be enabled Start kubelet on a cgroupv2 image with enabled.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /cc


Question: // If hints is nil, insert a single, preferred any-numa hint into allProviderHints. if len(hints) == 0 { klog.Infof("[topologymanager] Hint Provider has no preference for NUMA affinity with any resource") <del> allProviderHints = append(allProviderHints, []TopologyHint{{defaultAffinity, true}}) </del> <ins> allProviderHints = append(allProviderHints, []TopologyHint{{nil, true}}) </ins> continue }

What happened: I think I'm seeing a problem with the topology manager. I have two numa nodes, 22 cores per node. Kubelet has "--cpu-manager-policy=static --topology-manager-policy=single-numa-node ", and when I try to create a simple pod with 3 cpus and 500Mi of RAM in the guaranteed class it fails. Describing the pod gives "Resources cannot be allocated with Topology Locality", but I know there's lots of CPU and RAM available on both NUMA nodes. I'm instrumenting the topology manager kubelet code, currently there's not enough debug logs to actually be able to tell what's going wrong. What you expected to happen: I expect the pod container to be created on one or the other numa node and get allocated exclusive CPUs (via the static policy) on that NUMA node, and be affined to those CPUs. How to reproduce it (as minimally and precisely as possible): On a kubernetes node with multiple NUMA nodes, enable kubelet with static cpu manager policy and single-numa-node topology manager policy. Try to create a basic pod with a single container in the guaranteed QoS class. Anything else we need to know?: Environment: Kubernetes version (use ): sysadmin chris(keystoneadmin)]$ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"archive", BuildDate:"2019-10-09T05:41:44Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: wildcat pass hardware, 22 cores per NUMA node, two NUMA nodes. 64GB RAM per NUMA node. OS (e.g: ): CentOS 7 Kernel (e.g.

): 3.10.0-
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
oops, I thought the "close" meant the comment, not the whole issue. :)
I was able to reproduce this issue. The following (new) uint test in fails unexpectedly: With any policy other than the this test passes. This is due to 3 factors: 1) The CPUManager is the only hint provider passing actual valid hints 2) The devicemanager passes back as its hints, resulting in a single (i.e. don’t care) hint 3) We a special casing for the in the device manager that essentially turns the devicemanager's hint into a : Because the single hint from the devicemanager now has a preference, there is no chance to have it combine with any of the CPUManager hints to result in a successful hint. This should ultimately be addressed as part of the following in 1.17 But we should put a one-off patch into the 1.16.x branch to fix this in 1.16.
ptal
I've just updated with my observations. can you have a look and comment ?


Question: }) <del> ginkgo.It("should validate Replicaset Status endpoints", func() { </del> <ins> /*	Release: v1.22 Testname: ReplicaSet, status sub-resource Description: Create a ReplicaSet resource which MUST succeed. Attempt to read, update and patch its status sub-resource; all mutating sub-resource operations MUST be visible to subsequent reads. */ framework.ConformanceIt("should validate Replicaset Status endpoints", func() { </ins> testRSStatus(f) }) })

[X] test pr : kuberenetes/kubernetes 25 May [x] two weeks soak end date : 8 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints that are untested. Create a watch to track replica set events Create a replica set with a static label. Confirm that the pods are running. Get the replica set status. Parse the response and confirm that the replica set status conditions can be listed. Update the replica set status. Confirm via the watch that the status has been updated. Patch the replica set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for daemon set lifecycle test. Due to a test flake with patching the replica set status conditions the final “watch/validation” check is still be written. [It] should validate Replicaset Status endpoints Apr 6 15:16:30.093: INFO: labels: name=sample-pod,pod=httpd STEP: Create a ReplicaSet STEP: Verify that the required pods have come up. Apr 6 15:16:30.099: INFO: Pod name sample-pod: Found 0 pods out of 1 Apr 6 15:16:35.108: INFO: Pod name sample-pod: Found 1 pods out of 1 STEP: ensuring each pod is running STEP: Getting /status Apr 6 15:16:35.136: INFO: ReplicaSet test-rs has Conditions: [] Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: updating the ReplicaSet Status Apr 6 15:16:35.147: INFO: updatedStatus.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}},

Reason:"E2E", Message:"Set from e2e test"}} STEP: watching for the daemon set status to be updated Apr 6 15:16:35.149: INFO: Observed event: ADDED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Found replica set test-rs in namespace replicaset-2037 with labels: map[name:sample-pod pod:httpd] annotations: map[] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}] Apr 6 15:16:35.149: INFO: Replica set test-rs has an updated status STEP: get ReplicaSet state Apr 6 15:16:35.152: INFO: rs1.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Reason:"E2E", Message:"Set from e2e test"}} Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.FullyLabeledReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: patching the ReplicaSet Status Apr 6 15:16:35.158: FAIL: Failed to patch replica set status.

"test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] Unexpected error: <*errors.StatusError | 0xc00024d4a0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: " "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas]", Reason: "Invalid", Details: { Name: "test-rs", Group: "apps", Kind: "ReplicaSet", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.fullyLabeledReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.readyReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.availableReplicas", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] occurred The error looks to be part of the . Looking for suggestions on whats likely to be happening and how to best resolve the flake would be appreciated.

This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // same status. newStatus.ObservedGeneration = rs.Generation <del> var getErr error </del> <ins> var getErr, updateErr error var updatedRS *extensions.ReplicaSet </ins> for i, rs := 0, rs; ; i++ { <del> glog.V(4).Infof(fmt.Sprintf("Updating replica count for ReplicaSet: %s/%s, ", rs.Namespace, rs.Name) + </del> <ins> glog.V(4).Infof(fmt.Sprintf("Updating status for ReplicaSet: %s/%s, ", rs.Namespace, rs.Name) + </ins> fmt.Sprintf("replicas %d->%d (need %d), ", rs.Status.Replicas, newStatus.Replicas, *(rs.Spec.Replicas)) + fmt.Sprintf("fullyLabeledReplicas %d->%d, ", rs.Status.FullyLabeledReplicas, newStatus.FullyLabeledReplicas) + fmt.Sprintf("readyReplicas %d->%d, ", rs.Status.ReadyReplicas, newStatus.ReadyReplicas) +

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: } fakeDocker.containerList = []docker.APIContainers{ { <del> // the '--' is required for the kubelet to manage the container Names: []string{"foo--bar"}, </del> <ins> // the k8s prefix is required for the kubelet to manage the container Names: []string{"/k8s--foo--bar"}, </ins> ID:    "1234", }, { // network container <del> Names: []string{"k8snet--foo--"}, </del> <ins> Names: []string{"/k8s--net--foo--"}, </ins> ID:    "9876", }, {

release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)


Question: client := clientValue.Load().(*clientv3.Client) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() <del> // See https://github.com/etcd-io/etcd/blob/master/etcdctl/ctlv3/command/ep_command.go#L118 _, err := client.Get(ctx, path.Join(c.Prefix, "health")) </del> <ins> // See https://github.com/etcd-io/etcd/blob/c57f8b3af865d1b531b979889c602ba14377420e/etcdctl/ctlv3/command/ep_command.go#L118 _, err := client.Get(ctx, path.Join("/", c.Prefix, "health")) </ins> if err == nil { return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: apiserver fails to pass readiness/liveness check when we upgrade k8s 1.12 to k8s.14+. there are some detail error messages in apiserver when we increase log level to 6(healthz check etcd failed: error getting data from etcd: etcdserver: permission denied.) our etcd cluster grant specified prefix(for example,/cluster-1) permission to k8s cluster. there is no problem in k8s1.12. etcd healthz check is implemented by getting prefix+'/'+health path in k8s 1.14+,but prefix do not join '/'. there is no problem in etcd3 store, pathPrefix is joined by '/' and prefix. What you expected to happen: prefix joins '/' in etcd healthz check.
/sig api-machinery
/assign
GitHub didn't allow me to assign the following users: jingyih. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What custom etcd prefix are you passing in? if you prefix it with , that should resolve this issue for you in the short-term. I agree the health check key prefix should be computed the same way as the etcd client prefix
thanks. yes, we prefix it with '/' to resolve this issue.


Question: - apiGroups: ["extensions", "networking.k8s.io"] resources: ["ingresses"] verbs: ["get", "list", "watch"] <del> # For now, GLBC annotates ingress resources with various state and statuses: </del> <ins> # For now, GLBC annotates ingress resources with various state and statuses: </ins> # https://github.com/kubernetes/ingress-gce/blob/50d49b077d9ab4362a02fae05f94e433cd3f08dc/pkg/controller/controller.go#L579 # TODO(rramkumar1): Remove unnecessary `update` permission once statuses are propagated through `ingresses/status` - apiGroups: ["extensions", "networking.k8s.io"]

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which test(s) are failing: Since when has it been failing: The full testgrid history is red, but it has been failing at least since 5/29 Testgrid link: Reason for failure: Anything else we need to know: Originally these failures were due to an rbac permission issue (see: ), but now that that has been resolved, the tests are still not passing.
/sig network
adding to the milestone for investigation, since ingress was touched this release /milestone v1.19
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
what is the difference between that job and (which seems to be mostly passing)?
/assign
Also assigning since I think he's already looking into this: /assign
Oh nice, prow can actually assign me now
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Sorry for the delay. This has been triaged. The actual issue is that: system:controller:glbc rbac needs patch permissions for "ingresses/status" in I will send a PR for the fix. /remove-triage unresolved
^ PR
looks like that resolved almost all the failures. is still failing
Nice! Looks like this failure is caused by relying on the old static IP behavior. We used to allocate one if the specified one did not exist. I will send a PR to update the test.
can we close this out with ?
Yes, latest run was green:
/reopen Can this be reviewed again please? cc
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
seems to be a RBAC issue?
/cc did ingress-gce just start expecting access to this resource recently? does the RBAC role at need to be updated?
Hi, I've filed to address the issue.


Question: // the http.ResponseWriter. We can recover panics from go-restful, and // the logging value is questionable. type respLogger struct { <del> hijacked           bool statusRecorded     bool status             int statusStack        string </del> <ins> hijacked       bool statusRecorded bool status         int statusStack    string // mutex is used when accessing addedInfo and addedKeyValuePairs. // They can be modified by other goroutine when logging happens (in case of request timeout) mutex              sync.Mutex </ins> addedInfo          strings.Builder addedKeyValuePairs []interface{} startTime          time.Time captureErrorOutput bool <del> req *http.Request w   http.ResponseWriter </del> <ins> req       *http.Request userAgent string w         http.ResponseWriter </ins> logStacktracePred StacktracePred }

Apiserver crashed with stacktrace: Not to crash I was able to reproduce it by "slowing down" deleting headers in Authentication filter (): And making requests (around 5-10 requests to trigger this race condition): Crash was caused by race condition - one goroutine was modifying headers of requests (for example authentication filter). The other goroutine was trying to get UserAgent from request, which is also taken from request's headers map. This causes unrecoverable panic. <detailsAll versions, tested on 1.21 </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /priority important-soon
/triage accepted
Reopening - since we need to cherrypick it to previous releasses.
- friendly ping on opening cherrypicks
I've opened two cherry-picks:
One more for 1.20:
All merged - thanks!


Question: } controller := NewCertificateController( <ins> "test", </ins> client, informerFactory.Certificates().V1beta1().CertificateSigningRequests(), handler,

There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth


Question: {% set auth_path = "-auth_path=/var/lib/kubelet/kubernetes_auth" -%} <ins> {% set registry_qps = "-registry_qps=0.1" %} </ins> DAEMON_ARGS="{{daemon_args}} {{etcd_servers}} {{apiservers}} {{auth_path}} {{hostname_override}} {{address}} {{config}} --allow_privileged={{pillar['allow_privileged']}}"

introduces PullPolicies (PullAlways, PullNever, PullIfNotPresent), but by default, the policy is PullAlways, which bite docker repository again today. The related #google-containers IRC conversation can be found at: We should Introduce a new policy to cap the number of attempts. Maybe getting rid of PullAlways completely? cc/
I would like to change the default one to PullIfNoPresent for now. We can continue discussing the right policies.
Suggestions: PullIfNotPresent should be our default? PullAlways needs to be nerfed. Suggestion: max N pull attempts per pod config change, with <attempt#*10 second delay between each pull attempt, where N is a small number like 3 or 5.
I think both PullIfNotPresent and PullAlways (if we decide to still have it) should be nerfed.
Agreed, when "not present", we should behave the same as PullAlways, which should be nerfed.
Not finding another issue, so recording here for posterity: we really need a TTL on images to make PullIfNotPresent maximally useful.
So if a registry is down and the cluster restarts, pods will eventually give up (if we're nerfing pull always)? Seems like we should drastically reduce pull frequency after failures, but giving up means you can get into weird various scenarios. Esp if the registry is hosted in the cluster. Why not just back off?
I vote for some exponential back off. , Clayton Coleman wrote:
We think we do have backoff now, check and close it if so.