Question: allErrs = append(allErrs, field.Required(projPath.Child("path"), "")) } <del> allErrs = append(allErrs, validateLocalNonReservedPath(source.ClusterTrustBundle.Path, projPath.Child("path"))...) </del> <ins> allErrs = append(allErrs, ValidateLocalNonReservedPath(source.ClusterTrustBundle.Path, projPath.Child("path"))...) </ins> curPath := source.ClusterTrustBundle.Path if !allPaths.Has(curPath) {

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: } cfg.Profiles = completedProfiles <ins> if len(fileName) > 0 { // Since the default component config lists all the default plugins as enabled // without disabling them, we must explicitly disable all the plugins for each // extension point using the "*" expression to ensure that the generated config // file is usable disabledPlugins := []config.Plugin{{Name: "*"}} for i := range cfg.Profiles { if cfg.Profiles[i].Plugins == nil { continue } cfg.Profiles[i].Plugins.QueueSort.Disabled = disabledPlugins cfg.Profiles[i].Plugins.PreFilter.Disabled = disabledPlugins cfg.Profiles[i].Plugins.Filter.Disabled = disabledPlugins cfg.Profiles[i].Plugins.PostFilter.Disabled = disabledPlugins cfg.Profiles[i].Plugins.PreScore.Disabled = disabledPlugins cfg.Profiles[i].Plugins.Score.Disabled = disabledPlugins cfg.Profiles[i].Plugins.Reserve.Disabled = disabledPlugins cfg.Profiles[i].Plugins.Permit.Disabled = disabledPlugins cfg.Profiles[i].Plugins.PreBind.Disabled = disabledPlugins cfg.Profiles[i].Plugins.Bind.Disabled = disabledPlugins cfg.Profiles[i].Plugins.PostBind.Disabled = disabledPlugins } } </ins> buf, err := encodeConfig(cfg) if err != nil { return err

--write-to-config flag in the scheduler writes to a file the default component config; however this config if passed back to the scheduler is not usable because it lists all default plugins as enabled without disabling them. To make the configuration usable as an input config while at the same time useful for debugging, the output should disable all plugins for each extension point using the "*" expression. /help /sig scheduling /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : --write-to-config flag in the scheduler writes to a file the default component config; however this config if passed back to the scheduler is not usable because it lists all default plugins as enabled without disabling them. /help /sig scheduling /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: // NodeMetrics sets resource usage metrics of a node. type NodeMetrics struct { <del> metav1.TypeMeta   `json:",inline"` </del> <ins> metav1.TypeMeta `json:",inline"` // Standard object's metadata. // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata // +optional </ins> metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` // The following fields define time interval from which metrics were

no explanation for metadata filed of NodeMetrics/PodMetrics, code: should be like $ kubectl explain NodeMetrics KIND: NodeMetrics VERSION: DESCRIPTION: NodeMetrics sets resource usage metrics of a node. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. usage <map[string]string-required- The memory usage is the memory working set. window <string-required- and $ kubectl explain PodMetrics KIND: PodMetrics VERSION: DESCRIPTION: PodMetrics sets resource usage metrics of a pod. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: containers <[]Object-required- Metrics for all containers are collected within the same time window. kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. window <string-required- able to see the explanation for metadata filed of NodeMetrics/PodMetrics Kubernetes Version: v1.22.0-rc.0+
/sig instrumentation
/kind documentation /assign /triage accepted


Question: Check(req *http.Request) error } <del> var defaultHealthz = sync.Once{} // DefaultHealthz installs the default healthz check to the http.DefaultServeMux. func DefaultHealthz(checks ...HealthzChecker) { defaultHealthz.Do(func() { InstallHandler(http.DefaultServeMux, checks...) }) } </del> // PingHealthz returns true automatically when checked var PingHealthz HealthzChecker = ping{}

The debugging endpoint is exposed over the unauthenticated Kubelet healthz port. Versions prior to 1.15.0, 1.14.4, 1.13.8, and 1.12.10 are affected. The issue is of medium severity, but not exposed by the default configuration. If you are exposed we recommend upgrading to at least one of the versions listed. Am I vulnerable? By default, the Kubelet exposes unauthenticated healthz endpoints on port :, but only over localhost. If your nodes are using a non-localhost healthzBindAddress (--health-bind-address), and an older version, you may be vulnerable. If your nodes are using the default localhost healthzBindAddress, it is only exposed to pods or processes running in the host network namespace. Run to see whether nodes are running a vulnerable version. Run to check whether the "healthzBindAddress" is non-local. How do I mitigate the vulnerability? Upgrade to a patched version (1.15.0+, 1.14.4+, 1.13.8+, or 1.12.10+) or, update node configurations to set the "healthzBindAddress" to "127.0.0.1". fixed in 1.12.10 fixed in 1.13.8 fixed in 1.14.4 fixed in 1.15.0 Vulnerability Details The endpoint is exposed over the Kubelet's healthz port. This debugging endpoint can potentially leak sensitive information such as internal Kubelet memory addresses and configuration, or for limited denial of service. Thanks to Jordan Zebor of F5 Networks for reporting this problem. /area security /close
Closing this issue. <detailsIn response to : The debugging endpoint is exposed over the unauthenticated Kubelet healthz port. Versions prior to 1.15.0, 1.14.4, 1.13.8, and 1.12.10 are affected. The issue is of medium severity, but not exposed by the default configuration. If you are exposed we recommend upgrading to at least one of the versions listed. By default, the Kubelet exposes unauthenticated healthz endpoints on port :, but only over localhost.

If your nodes are using a non-localhost healthzBindAddress (--health-bind-address), and an older version, you may be vulnerable. If your nodes are using the default localhost healthzBindAddress, it is only exposed to pods or processes running in the host network namespace. fixed in 1.13.8 fixed in 1.14.4 fixed in 1.15.0 The endpoint is exposed over the Kubelet's healthz port. This debugging endpoint can potentially leak sensitive information such as internal Kubelet memory addresses and configuration, or for limited denial of service. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/label official-cve-feed (Related to )


Question: case *metav1beta1.Table: includesTable = true <del> if err := NewTableSorter(t, r.field).Sort(); err != nil { continue </del> <ins> if sorter, err := NewTableSorter(t, r.field); err != nil { return err } else if err := sorter.Sort(); err != nil { return err </ins> } default: includesRuntimeObjs = true

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: seems to have regressed in v1.13.0 and no longer sorts by creation timestamp. What you expected to happen: should sort by creation timestamp. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: N/A Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
/assign
bisected to () /assign
opened to fix
/assign


Question: - apiGroups: ["cloud.google.com"] resources: ["backendconfigs"] verbs: ["get", "list", "watch", "update", "create", "patch"] <ins> - apiGroups: ["networking.gke.io"] resources: ["servicenetworkendpointgroups"] verbs: ["get", "list", "watch", "update", "create", "patch"] </ins>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which test(s) are failing: Since when has it been failing: The full testgrid history is red, but it has been failing at least since 5/29 Testgrid link: Reason for failure: Anything else we need to know: Originally these failures were due to an rbac permission issue (see: ), but now that that has been resolved, the tests are still not passing.
/sig network
adding to the milestone for investigation, since ingress was touched this release /milestone v1.19
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
what is the difference between that job and (which seems to be mostly passing)?
/assign
Also assigning since I think he's already looking into this: /assign
Oh nice, prow can actually assign me now
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Sorry for the delay. This has been triaged. The actual issue is that: system:controller:glbc rbac needs patch permissions for "ingresses/status" in I will send a PR for the fix. /remove-triage unresolved
^ PR
looks like that resolved almost all the failures. is still failing
Nice! Looks like this failure is caused by relying on the old static IP behavior. We used to allocate one if the specified one did not exist. I will send a PR to update the test.
can we close this out with ?
Yes, latest run was green:
/reopen Can this be reviewed again please? cc
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
seems to be a RBAC issue?
/cc did ingress-gce just start expecting access to this resource recently? does the RBAC role at need to be updated?
Hi, I've filed to address the issue.


Question: case status.State.Running != nil: return false, fmt.Errorf("subpath container unexpectedly became running") case status.State.Terminated != nil: <ins> if status.State.Terminated.ExitCode != 0 && allowContainerTerminationError { return true, nil } </ins> return false, fmt.Errorf("subpath container unexpectedly terminated") case status.State.Waiting != nil: if status.State.Waiting.Reason == "CreateContainerConfigError" &&

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The read only sig-storage tests validate that Pods can come up when readOnly is set. But it doesn't actually validate that the container cannot write to the volume. What you expected to happen: Tests should validate that the container cannot write to readonly volumes. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can this issue be closed after your last PR or is there more to work on? If yes I'd like to know what's left because I'm looking for something to work on
I think we should actually reinstate the test case I removed and enhance it to handle the two scenarios I mentioned. It would also be good to see if we have other readOnly test cases that could be enhanced as well
Thanks ! I have a few questions: 1) how do I run the tests in locally? and 2) how can I assert writing to volume has failed? Thank you in advance!!


Question: } func newController(nodeNames []string, batchPeriod time.Duration) (*fake.Clientset, *endpointSliceController) { <del> client := newClientset() </del> <ins> client := fake.NewSimpleClientset() </ins> informerFactory := informers.NewSharedInformerFactory(client, controller.NoResyncPeriodFunc()) nodeInformer := informerFactory.Core().V1().Nodes() indexer := nodeInformer.Informer().GetIndexer()

The controllers can create a second EndpointSlice after an immediate sync if the first EndpointSlice they created is not in the cache yet. Here's what can happen: gets event that triggers sync for Service syncs and creates EndpointSlice gets event that triggers sync for same Service syncs and creates EndpointSlice (the slice created in 2 has not made it to the cache yet) A single EndpointSlice to be created. Run integration tests hundreds of times that involve 2 consecutive events that will trigger a sync. Example: /assign /sig network
/triage accepted
/cc
When this bug is fixed, we'll need to rollback
just amend it in the PR that fix the bug, so we merge all together


Question: nodeMonitorGracePeriod time.Duration, nodeStartupGracePeriod time.Duration, nodeMonitorPeriod time.Duration) *NodeController { <ins> eventBroadcaster := record.NewBroadcaster() recorder := eventBroadcaster.NewRecorder(api.EventSource{Component: "controllermanager"}) if kubeClient != nil { glog.Infof("Sending events to api server.") eventBroadcaster.StartRecordingToSink(kubeClient.Events("")) } else { glog.Infof("No api server defined - no events will be sent to API server.") } </ins> return &NodeController{ cloud:                   cloud, matchRE:                 matchRE,

/cc
Is this from head? I've never seen this before, maybe changed anything recently. Anyway, the code in question will be deleted by
I saw this in the Travis run for an unrelated PR (Kubelet test refactoring). If it's going away, then that's ho we'll fix it :)
Yup. It'll go away. I'll assign it to myself - I'll run test few times after to make sure it's gone.
another example: https://travis-
Just hit this again: https://travis-
I was hoping we could finish getting rid of this code this week, which is why I had been ignoring this. But we ran out of time and Poland has a holiday on Monday, so I guess we need to fix this race. I'll take a look.
Thanks. This is becoming a real nuisance that is masking other failures.
To repro I assume I just type the commands from Travis, e.g. the shell stuff in https://travis- ?
Never mind, I got it. Though the first time I ran it all the tests passed.
I don't think the failure is deterministic, but yes.
This is super annoying, it happens way too much of the time...
Sigh, I was hoping we can get rid of the code this week. I think the problem comes from someone changes the event recording. I'll send a PR
It seems to be basically always failing now, except when some other flake fails first.
It would be fine with me if we just temporarily disabled event generation there.
If I'm reading the stack traces correctly, what's going on is that NewNodeController [] calls StartRecordingToSink() [] which calls StartEventWatcher() which calls eventHandler() which calls StartRecordingToSink() which calls sink.Create() which is Create() in , which appends to c.Fake.Actions. At the same time, the last line of the test tries to print "item.fakeNodeHandler.Actions" and these are presumably the same variable. I'm not sure why these are running at the same time, though.
+1 to Braians idea to temporary disable event generation.
Re-titled issue.
Re-titled issue.


Question: } notMnt, mntErr := mounter.IsLikelyNotMountPoint(mountPath) if mntErr != nil { <del> return err </del> <ins> return mntErr </ins> } if notMnt { glog.V(4).Infof("%q is unmounted, deleting the directory", mountPath)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.

What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: &HorizontalPodAutoscalerList{}, &Scale{}, &v1.ListOptions{}, <ins> &v1.DeleteOptions{}, </ins> ) versionedwatch.AddToGroupVersion(scheme, SchemeGroupVersion) }

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form:


Question: operator: "Exists" containers: - name: coredns <del> image: coredns/coredns:0.9.10 </del> <ins> image: coredns/coredns:1.0.1 </ins> imagePullPolicy: IfNotPresent resources: limits:

Is this a BUG REPORT or FEATURE REQUEST?: BUG What happened: When CoreDNS is deployed by default using feature-gates of kubeadm, it does not log anything when processing a DNS query. That is due to change of log option of CoreDNS in latest version. that version was bumped up in kubeadm, but the manifest was not adapted to the new log option. initial change of 'log stdout' in CoreDNS : version of CoreDNS 1.0.0 bumped into Kubeadm : NOTE: I guess we should also changed manifests for kube-up : see The change in manifest expected is to remove the option "stdout" in the line "log ...". As it is now interpreted as a scope of domain to log. Invalid manifest: Environment: Kubernetes version (use ): 1.9 and MAIN Cloud provider or hardware configuration: any Install tools: kubeadm, kube-up the configmap for coredns kubectl get configmaps -n kube-system coredns -o yaml kubectl replace -f - the coredns deployment kubectl scale deployment --replicas=0 -n kube-system coredns kubectl scale deployment --replicas=1 -n kube-system coredns
/cc /sig network
NOTE: only log is impacted. CoreDNS working as expected.
/cc
I have a question : the initial manifest is done for the image 0.9.10 of CoreDNS. Bumping the version of CoreDNS was done by changing the version in file (see ) which means that the image's version is STILL 0.9.10 in the manifests ... and the corefile config is correct against the version of the manifest. QUESTION: when modifying the , is there a way to upgrade the manifest by program ? I guess at the time the new version number is applied on the manifest ...
thanks for reporting this! I send PR to update the Corefile. I don't think of a way so far. I guess we can't unless there is a way to get the Corefile from some place of upstream (CoreDNS).


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package queueset import ( "testing" ) func TestGetNextFinish(t *testing.T) { var virtualStart float64 = 100 var G float64 = 60 tests := []struct { name                  string requests              []*request virtualFinishExpected float64 }{ { name: "for the oldest request", requests: []*request{ {width: 5}, {width: 6}, {width: 7}, }, virtualFinishExpected: virtualStart + (5 * G), }, { name:                  "queue does not have any request waiting", requests:              []*request{}, virtualFinishExpected: virtualStart, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { requests := newRequestFIFO() for i := range test.requests { requests.Enqueue(test.requests[i]) } q := &queue{requests: requests} q.virtualStart = virtualStart virtualFinishGot := q.GetNextFinish(G) if test.virtualFinishExpected != virtualFinishGot { t.Errorf("Expected virtual finish time: %.9fs, but got: %.9fs", test.virtualFinishExpected, virtualFinishGot) } }) } } </ins>

I reviewed the current code to see what had been done to generalize request width. I found a few inconsistencies in . A really simple one is at line , where is subtracted from the product but should have been subtracted from just (and their difference multiplied by ). With that fixed, a queue's tracks the expected "virtual time" ( in the original paper) when the oldest waiting request will start. However, computes the expected finish "virtual time" of the J'th request by multiplying by but not . There should be a consistent notion of "virtual time" (AKA ), either always including seats or excluding them. Kubernetes version (use ): latest k8s/apiserver sources Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
BTW, I suspect it would be better if "virtualTime" is not multiplied by request width. I am working on writing.
/triage accepted
I am running into a dilemma. The whole point of "virtual time" --- R(t) in the original paper --- is that it is shared among all the queues. They all get work done at the same rate, dR/dt. But when we generalize so that requests have different widths (and thus work is the sum over requests of ), even restricting the virtual world so that a queue runs only one request at a time does not save us from the fact that different queues get work done at different rates. Thus, even discarding fairness, and executing one request at a time per queue in the virtual world, it looks like what can be shared among queues is not a matter of work done.
But the current implementation approach makes the assumption that all the non-empty queues get work done at the same rate. As long as we are willing to tolerate the lossages due to that fiction, the queues can share one R(t) meter.
I see both sides of this; I still think it is double counting to use to advance virtual time AND to occupy W seats. OTOH, I also think it is right that we should be fair over the work () -- not just duration. One crazy idea is to add the request to multiple queues, and not execute it until it gets to the head of all the queues it is in.

The number of queues it needs to be to would be (# queues) * W / seats, rounded up I guess.
I think "virtual time" is a fuzzy concept that does not come equipped with guidelines about how to think about it correctly. That is why I followed the style of reasoning in the original paper, it defines a specific function and shows why and how that particular function is useful. In the KEP revision I am preparing, I think the problem statement ()the equations defining intended behavior ()the implementation discussion () make sense. It is in the last of those that is introduced, and I think it is clear that this should be a measure that includes request width.
I don't think it's double counting, because those are different things. And yes - we should be fair over WD - this is exactly why I wanted to push virtual time by WD. - I've seen your KEP PR, just I'm under water. My Fridays are real working time, so I will take a look tomorrow.


Question: // Note that this is annoying: the load balancer disappears from the API immediately, but it is still // deleting in the background.  We get a DependencyViolation until the load balancer has deleted itself <ins> var loadBalancerSGs = aws.StringValueSlice(lb.SecurityGroups) describeRequest := &ec2.DescribeSecurityGroupsInput{} filters := []*ec2.Filter{ newEc2Filter("group-id", loadBalancerSGs...), } describeRequest.Filters = c.tagging.addFilters(filters) response, err := c.ec2.DescribeSecurityGroups(describeRequest) if err != nil { return fmt.Errorf("error querying security groups for ELB: %q", err) } </ins> // Collect the security groups to delete securityGroupIDs := map[string]struct{}{} <del> for _, securityGroupID := range lb.SecurityGroups { if *securityGroupID == c.cfg.Global.ElbSecurityGroup { //We don't want to delete a security group that was defined in the Cloud Configurationn. </del> <ins> for _, sg := range response { sgID := aws.StringValue(sg.GroupId) if sgID == c.cfg.Global.ElbSecurityGroup { //We don't want to delete a security group that was defined in the Cloud Configuration. </ins> continue } <del> if aws.StringValue(securityGroupID) == "" { klog.Warning("Ignoring empty security group in ", service.Name) </del> <ins> if sgID == "" { klog.Warningf("Ignoring empty security group in %s", service.Name) </ins> continue } <del> securityGroupIDs[*securityGroupID] = struct{}{} </del> <ins> if !c.tagging.hasClusterTag(sg.Tags) { klog.Warningf("Ignoring security group with no cluster tag in %s", service.Name) continue } securityGroupIDs[sgID] = struct{}{} </ins> } // Loop through and try to delete them

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When specifying multiple services using the annotation the SG defined in that annotation may be the one used for ingress to the instances. If this happens for multiple service ELBs and one of those services is removed then the rule may be removed during clean up causing the other ELBs losing the ability to contact their instances. What you expected to happen: Removal of any service should not lead to disruption of other services. How to reproduce it (as minimally and precisely as possible): a security group in AWS. multiple services referencing that SG in . one of those services. Environment: Kubernetes version (use ): 1.8.8 Cloud provider or hardware configuration: AWS Kernel (e.g. ): 4.4.121-k8s
From what I can tell this is a result of the simple selection of what to consider the LB security group here: I'd think a solution would be to prefer an SG tagged as kube owned if one is present (which there should always be if one is created for the LB as expected). I should be able to get a PR together if that sounds like an appropriate fix.
This may also be related to
/sig aws
Also
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have a Kubernetes () in AWS. I created a Service of type and used the annotation to apply an extra security group to the ELB that gets created. The extra security group was created outside of Kubernetes, with the expectation that it is not owned by Kubernetes and will be managed independently. I deleted the Service resource. Kubernetes then deleted the ELB and my extra security group. What you expected to happen: I expect my extra security group to not be deleted. How to reproduce it (as minimally and precisely as possible): In AWS, create security group that is otherwise unused. In Kubernetes, create Service of type , include the annotation with the previously created security group. Watch an ELB be created with that security group. Delete the Service from Kubernetes. Watch the ELB be deleted (as expected) and the security group be deleted (unexpected). Anything else we need to know?: It seems a workaround may be to ensure the security group is in use by at least one other resource within AWS so that the deletion attempt will fail with a . The deletion process will eventually timeout, if I understand the code correctly. I've linked to , though I don't see any additional behavior on , so I believe the bug is there as well. Environment: Kubernetes version (use ): v1.7.16 Cloud provider or hardware configuration: AWS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have also experienced this same issue luckily we have a very strict iam policy for k8s in aws and did not give the controller access to delete the 'extra-security-group` we specified in the annotation. There also seems to not be a try/catch around deleting security groups when deleting ELBs.

The flow looks like the following: 1) Fetch security groups associated with ELB 2) For each security group try to delete the security group, if the security group fails to delete do not move on to the next security group to delete In this scenario the security group that is created by k8s for this ELB does not get deleted and is left behind. Here are some relevant logs, is the k8s generated sg is the security group specified in the annotation We are using version
I believe the solution should be to only have k8s try to delete security groups that it has ownership tags for.
Another solution could be while to "delete" subtract the set from the annotation aws-load-balancer-extra-security-groups
A "workaround" for this issue is to add the SG as a source in the nodes SG
I'm hitting the same issue on this one and it's causing cluster networking issues - Kubernetes deletes a shared SG which breaks my API access - I have the shared ownership tag to the security group but it is still removed. Any ideas?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
This issue is affecting me as well.


Question: vm, err := getVirtualMachineByName(i.cfg, ctx, c, name) if err != nil { <ins> if _, ok := err.(*find.NotFoundError); ok { return "", cloudprovider.InstanceNotFound } </ins> return "", err }

ExternalID/InstanceID do not return 'cloudprovider.InstanceNotFound' error when the VM is not found. This can occur when a node VM is deleted entirely. In this case InstanceID method would throw an nil pointer exception here: since returned error here: is not being handled, vm would be nil. vm object is then dereferenced in getVirtualMachineManagedObjectReference func. In ExternalID method, the error is being handled by just passing it up. Node controller then prints out error and continues: e.g. ExternalID/InstanceID should handle the 'nod found' error being returned from vCenter and return 'cloudprovider.InstanceNotFound'.


Question: import ( "testing" <ins> "github.com/stretchr/testify/assert" </ins> discovery "k8s.io/api/discovery/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> ) func TestEndpointSliceTrackerUpdate(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: # of the files to be packaged.  This assumes that ${2}/kubernetes is what is # being packaged. function kube::release::create_tarball() { <ins> kube::build::ensure_tar </ins> local tarfile=$1 local stagingdir=$2 <del> # Find gnu tar if it is available local tar=tar if which gtar &>/dev/null; then tar=gtar else if which gnutar &>/dev/null; then tar=gnutar fi fi local tar_cmd=("$tar" "czf" "${tarfile}" "-C" "${stagingdir}" "kubernetes") if "$tar" --version | grep -q GNU; then tar_cmd=("${tar_cmd[@]}" "--owner=0" "--group=0") else echo "  !!! GNU tar not available.  User names will be embedded in output and" echo "      release tars are not official. Build on Linux or install GNU tar" echo "      on Mac OS X (brew install gnu-tar)" fi "${tar_cmd[@]}" </del> <ins> "${TAR}" czf "${tarfile}" -C "${stagingdir}" kubernetes --owner=0 --group=0 </ins> } # ---------------------------------------------------------------------------

Since I've been having troubles with installation, I've digged a bit and found out that the permissions inside Kubernetes tarballs changed between 0.16.0 and 0.17.0. I've downloaded both tarballs, extracted them into separate directories and then extracted the Kubernetes server tarballs. This is what I get: I suppose, there has been a concious decision to restrict the world's access to those binaries. However, this breaks the installation routine mentioned above and probably makes life a bit more difficult in several other places. I'd be thankful for a correction if this wasn't intended or a pointer to the appropriate commit it this is legit -- I'd correct the installation.
assigning this to you to decide who the eventual assignee should be :)
Any news on this front? 0.20.1 is still restricted.
Apologies for the issue. It wasn't intentional -- likely an artifact of build scripts being run by different people? I'll check out 0.20.2 as I build it to make sure it has the right perms.
Apologies, but 0.20.2 didn't include fixed permissions. I now understand the issue, : "make release" builds a tarball using the permissions on files in your local git tree. I had "umask 027" set when I cloned, so those in-the-repo files had no world-readable perms. Setting "umask 022" before running "make release" is not enough -- it has to be part of creating the local tree when cloning. I'll update instructions and the next release (20.3 or 21.0) should fix this issue. And longer term, we need to put a bit more effort into releasing from a known-good environment and not rely on a developer's workstation's idiosyncrasies -- sorry this bit you.
I can't repro Trying again a different way, but I don't think the umask before clone should do anything (in particular, umasks aren't "sticky" on files, they're sticky to processes.)
And let's pretend on my personal fork isn't woefully out of date (who uses master on their personal fork? I should really just delete that that thing.) Comment updated.
Ah, I see the problem now.

works fine for the server tar, but the full tar also packages examples, etc., and for that, tar tends to just pick up the permissions of the git tree. Got it.
I'll take this. I'm going to write a script to build official bits, and this is easy enough to fix along the way.
Seems not fixed as of Kubernetes 1.0.7
sorry about that. is doing the 1.1.1 release and has been improving release scripts -- is 1.1.1 good re: permissions?
was not built by .
Reassigning to as release-oncall. Quinton, I can mentor or advise as necessary.
And yeah, is fine: There's no real bug here. There was a process issue that occurred with 1.0.7. We can rebuild the binaries if we need to.
FYI, leaving this issue closed, as per 's comment above. If anyone disagrees, please re-open this issue.


Question: if err != nil { return []*v1.Pod{}, err } <del> // TODO: Do we need to copy here? result := make([]*v1.Pod, 0, len(pods)) for i := range pods { result = append(result, &(*pods[i])) } return result, nil </del> <ins> return pods, nil </ins> } func (dc *DisruptionController) worker() {

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DisruptionController getPodsForPdb Is this a BUG REPORT or FEATURE REQUEST? FEATURE REQUEST in we have function return pods using . but at the end we perform a copy of the pod array that we listed with : this is not necessary cc
/assign
/kind enhancement


Question: func (a *azureDiskAttacher) WaitForAttach(spec *volume.Spec, devicePath string, _ *v1.Pod, timeout time.Duration) (string, error) { var err error <del> lun, err := strconv.Atoi(devicePath) </del> <ins> volumeSource, err := getVolumeSource(spec) </ins> if err != nil { <del> return "", fmt.Errorf("azureDisk - Wait for attach expect device path as a lun number, instead got: %s (%v)", devicePath, err) </del> <ins> return "", err </ins> } <del> volumeSource, err := getVolumeSource(spec) </del> <ins> diskController, err := getDiskController(a.plugin.host) </ins> if err != nil { return "", err } <ins> nodeName := types.NodeName(a.plugin.host.GetHostName()) diskName := volumeSource.DiskName glog.V(5).Infof("azureDisk - WaitForAttach: begin to GetDiskLun by diskName(%s), DataDiskURI(%s), nodeName(%s), devicePath(%s)", diskName, volumeSource.DataDiskURI, nodeName, devicePath) lun, err := diskController.GetDiskLun(diskName, volumeSource.DataDiskURI, nodeName) if err != nil { return "", err } glog.V(5).Infof("azureDisk - WaitForAttach: GetDiskLun succeeded, got lun(%v)", lun) </ins> exec := a.plugin.host.GetExec(a.plugin.GetPluginName()) io := &osIOHandler{} scsiHostRescan(io, exec) <del> diskName := volumeSource.DiskName nodeName := a.plugin.host.GetHostName() </del> newDevicePath := "" err = wait.Poll(1*time.Second, timeout, func() (bool, error) { <del> if newDevicePath, err = findDiskByLun(lun, io, exec); err != nil { </del> <ins> if newDevicePath, err = findDiskByLun(int(lun), io, exec); err != nil { </ins> return false, fmt.Errorf("azureDisk - WaitForAttach ticker failed node (%s) disk (%s) lun(%v) err(%s)", nodeName, diskName, lun, err) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This was introduced by following code change introduced in v1.10.0: candidate fix is here(only works on Linux, not a good solution): This issue is caused by PR: Environment: Kubernetes version (use ): v1.10 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Update: above fix does not work on Windows, I am still investigating a better solution. This issue is caused by updated after , related code is here: while should not be changed from LUN to real device path in azuredd since AttachDisk happens on master, is a LUN(Logical Unit Number) number after AttachDisk process in the beginning happens on agent node: on Linux node, could be ; while on Windows node, would be a disk number, when is recalled, it could not tell whether it's a LUN or a disk number (this is the key point).
thanks for issuing this problem. Could you submit a PR with your fix so that we can review it? Thanks!
finally I realized should not be used in func any more by referencing the gcepd code, PTAL , awsebs code should have same issue, I filed a aws_ebs issue here:


Question: }) /* <ins> Release: v1.25 Testname: Jobs, manage lifecycle </ins> Description: Attempt to create a suspended Job which MUST succeed. Attempt to patch the Job to include a new label which MUST succeed. The label MUST be found. Attempt to replace the Job to include a

[x] Test approval issue : [x] Test PR : - Needed an update on the handling of anotations 27 March 2022 [x] Two weeks soak end date : 6 April 2022 [x] Test promotion PR : [x] Update Test PR: 15 April 2022 [x] Two weeks soak end date : 4 May 2022 with start of 1.25 [x] Update Test Promotion: According to these APIsnoop queries, there are still four remaining Job endpoints which are untested. Using a number of existing e2e test practices a new has been created for this job lifecycle test. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for tracking e2e testgrid and promotion PR
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ReadPrivateKeyFromPEM is a helper function for reading a private key from a PEM-encoded file func ReadPrivateKeyFromPEM(data []byte) (interface{}, error) { <del> if key, err := jwt.ParseRSAPrivateKeyFromPEM(data); err == nil { return key, nil } if key, err := jwt.ParseECPrivateKeyFromPEM(data); err == nil { return key, nil </del> <ins> var block *pem.Block for { // read the next block block, data = pem.Decode(data) if block == nil { break } // get PEM bytes for just this block blockData := pem.EncodeToMemory(block) if key, err := jwt.ParseRSAPrivateKeyFromPEM(blockData); err == nil { return key, nil } if key, err := jwt.ParseECPrivateKeyFromPEM(blockData); err == nil { return key, nil } // tolerate non-key PEM blocks for compatibility with things like "EC PARAMETERS" blocks // originally, only the first PEM block was parsed and expected to be a key block </ins> } return nil, fmt.Errorf("data does not contain a valid RSA or ECDSA private key") }

Kubernetes version Serverversion 1.5.2 (running Hyperkube v1.5.2coreos.1 ()) Environment: Bare-Metal Coreos 1235.6.0 (Kernel 4.7.3-coreos-r2) What happened: kube-controller-manager with ecdsa key fails to start F0127 09:59:04. 1 ] error running controllers: Error reading key for service account token controller: error reading private key file data does not contain a valid RSA or ECDSA private key The key is valid (and works with e.g. the api-server) Command used to create the key:
the issue is the block. removing that (invoke openssl with to omit it from the start) will allow the key to be consumed. will fix in 1.6 to ignore that block when reading the private key block


Question: var _ = SIGDescribe("Deployment", func() { var ns string var c clientset.Interface <ins> var dc dynamic.Interface </ins> ginkgo.AfterEach(func() { failureTrap(c, ns)

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: }) framework.ExpectNoError(err) } <ins> func testRSScaleSubresources(f *framework.Framework) { ns := f.Namespace.Name c := f.ClientSet // Create webserver pods. rsPodLabels := map[string]string{ "name": "sample-pod", "pod":  WebserverImageName, } rsName := "test-rs" replicas := int32(1) ginkgo.By(fmt.Sprintf("Creating replica set %q that asks for more than the allowed pod quota", rsName)) rs := newRS(rsName, replicas, rsPodLabels, WebserverImageName, WebserverImage, nil) _, err := c.AppsV1().ReplicaSets(ns).Create(context.TODO(), rs, metav1.CreateOptions{}) framework.ExpectNoError(err) // Verify that the required pods have come up. err = e2epod.VerifyPodsRunning(c, ns, "sample-pod", false, replicas) framework.ExpectNoError(err, "error in waiting for pods to come up: %s", err) ginkgo.By("getting scale subresource") scale, err := c.AppsV1().ReplicaSets(ns).GetScale(context.TODO(), rsName, metav1.GetOptions{}) if err != nil { framework.Failf("Failed to get scale subresource: %v", err) } framework.ExpectEqual(scale.Spec.Replicas, int32(1)) framework.ExpectEqual(scale.Status.Replicas, int32(1)) ginkgo.By("updating a scale subresource") scale.ResourceVersion = "" // indicate the scale update should be unconditional scale.Spec.Replicas = 2 scaleResult, err := c.AppsV1().ReplicaSets(ns).UpdateScale(context.TODO(), rsName, scale, metav1.UpdateOptions{}) if err != nil { framework.Failf("Failed to put scale subresource: %v", err) } framework.ExpectEqual(scaleResult.Spec.Replicas, int32(2)) ginkgo.By("verifying the replicaset Spec.Replicas was modified") rs, err = c.AppsV1().ReplicaSets(ns).Get(context.TODO(), rsName, metav1.GetOptions{}) if err != nil { framework.Failf("Failed to get statefulset resource: %v", err) } framework.ExpectEqual(*(rs.Spec.Replicas), int32(2)) ginkgo.By("Patch a scale subresource") scale.ResourceVersion = "" // indicate the scale update should be unconditional scale.Spec.Replicas = 4    // should be 2 after "UpdateScale" operation, now Patch to 4 rsScalePatchPayload, err := json.Marshal(autoscalingv1.Scale{ Spec: autoscalingv1.ScaleSpec{ Replicas: scale.Spec.Replicas, }, }) framework.ExpectNoError(err, "Could not Marshal JSON for patch payload") _, err = c.AppsV1().ReplicaSets(ns).Patch(context.TODO(), rsName, types.StrategicMergePatchType, []byte(rsScalePatchPayload), metav1.PatchOptions{}, "scale") framework.ExpectNoError(err, "Failed to patch replicaset: %v", err) rs, err = c.AppsV1().ReplicaSets(ns).Get(context.TODO(), rsName, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get replicaset resource: %v", err) framework.ExpectEqual(*(rs.Spec.Replicas), int32(4), "replicaset should have 4 replicas") } </ins>

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a ReplicaSet with a static label Read the ReplicaSetScale Update the ReplicaSetScale Get the ReplicaSetScale to ensure it is updated Patch the ReplicaSetScale Read the ReplicaSetScale to ensure it is Patch Delete the Namespace and ReplicaSet Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource readAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource replaceAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture
/triage accepted
Presented in Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ${image_service_endpoint_args}  --port="$KUBELET_PORT" >"${KUBELET_LOG}" 2>&1 & KUBELET_PID=$! <ins> # Quick check that kubelet is running. if ps -p $KUBELET_PID > /dev/null ; then echo "kubelet ( $KUBELET_PID ) is running." else cat ${KUBELET_LOG} ; exit 1 fi </ins> else # Docker won't run a container with a cidfile (container id file) # unless that file does not already exist; clean up an existing

Problem Local-up-cluster will start normally even if the hyperkube kubelet startup command fails. Solution Ensure local-up-cluster fail-fast if things are broke or if individual daemons dont start


Question: v1.ResourceMemory: *resource.NewQuantity(400, resource.DecimalSI)}, } <del> // Create two pods. </del> <ins> // First pod will go waiting. </ins> waitingPod := initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "waiting-pod", Namespace: testCtx.NS.Name, Priority: &lowPriority, Resources: &resourceRequest}) waitingPod.Spec.TerminationGracePeriodSeconds = new(int64) waitingPod, err = createPausePod(testCtx.ClientSet, waitingPod)

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose?

I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.


Question: netutils "k8s.io/utils/net" ) <ins> const ( UnprivilegedUserToken = "unprivileged-user" ) </ins> // Config is a struct of configuration directives for NewControlPlaneComponents. type Config struct { // If nil, a default is used, partially filled configs will not get populated.

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted


Question: package apps_test import ( <del> "testing" </del> . "github.com/onsi/ginkgo" <ins> . "github.com/onsi/ginkgo/config" . "github.com/onsi/ginkgo/types" </ins> . "github.com/onsi/gomega" <ins> "fmt" "testing" </ins> ) func TestApps(t *testing.T) { RegisterFailHandler(Fail) <del> RunSpecs(t, "Apps Suite") </del> <ins> RunSpecsWithDefaultAndCustomReporters(t, "Apps Suite", []Reporter{newlineReporter{}}) </ins> } <ins> // Print a newline after the default newlineReporter due to issue // https://github.com/jstemmer/go-junit-report/issues/31 type newlineReporter struct{} func (newlineReporter) SpecSuiteWillBegin(config GinkgoConfigType, summary *SuiteSummary) {} func (newlineReporter) BeforeSuiteDidRun(setupSummary *SetupSummary) {} func (newlineReporter) AfterSuiteDidRun(setupSummary *SetupSummary) {} func (newlineReporter) SpecWillRun(specSummary *SpecSummary) {} func (newlineReporter) SpecDidComplete(specSummary *SpecSummary) {} // SpecSuiteDidEnd Prints a newline between "35 Passed | 0 Failed | 0 Pending | 0 Skipped" and "--- PASS:" func (newlineReporter) SpecSuiteDidEnd(summary *SuiteSummary) { fmt.Printf("n") } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind flake What happened: TestApps always failed Anything else we need to know?: https://k8s- https://k8s- /sig testing
my pr appear same problem. i will investigate the cause later.
/sig cli
/assign
GitHub didn't allow me to assign the following users: zjj2wry. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fails when run by . Stress testing runs 1000 times locally without any errors. https://k8s-
Hit this again here https://k8s-
this is odd... the test output actually shows TestApps succeeding in that run:
Happened again: https://k8s- The messages were interleaved . was appended at the end. I think that threw off.
By the looks of test grid, it looks like it's been failing consistently in both presubmit and postsubmit builds: https://k8s- https://k8s- The logs are happy, the junit output is not. Maybe it has something to do with the fact that ginkgo is used to run the test cases? It's used for TestOpenapi tests as well. There was an issue on those tests failing for what looks like the same reason Looks like they were fixed by /kind bug /priority failing-test
cc
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 6 days, the issue will be moved out of the v1.8 milestone. priority: Must specify exactly one of [, , ]. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: nil, true, 0, 0, 0, 0, 10, 0, 10, 0, 0, nil, "", }, <del> "to many job sync failure": { </del> <ins> "too many job failures": { </ins> 2, 5, 0, true, 0, nil, true, 0, 0, 0, 1, 0, 0, 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a job with and , the pod with the configuration included below continued to restart and was never marked as failed: What you expected to happen: We expected only 2 attempted pod restarts before the job was marked as failed. The pod kept restarting and was never set to failed, remained active. How to reproduce it (as minimally and precisely as possible): Create the above job and observe the number of pod restarts. Anything else we need to know?: The flag works as expected when . Environment: Kubernetes version (use ): 1.8.0 using minikube
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this is because for , kubelet will try to restart failed pod in place, and the pod status will be even the container is failed to run, so jobcontroller get a illusion as the pod is still running. Here's my naive thoughts: for short-term to avoid this, add a , for long-term we need consider how to recognize and count this kind of Pod as failed pod. cc
Just ran into the issue as well. It might be useful to amend the documentation to warn about the fact that won't be effective if is chosen as restart policy. At least, it wasn't obvious to me and I didn't catch any relevant note in the docs. Happy to do a quick PR if that's a desirable short-term action. For Pods failing as reported in this issue, what's the difference in behavior between a restart policy and a hypothetical scenario where is respected by ? Anything other than failing Pods being replaced by newly created ones in the former case vs. existing Pods being restarted in the latter?
For OnFailure, the failed Pod will restart on the same kubelet host, so emptyDir/hostPath is still there, for Never the failed Pod got delete and a replacement will create on another(may be the same) kubelet host, if the Pod use emptyDir/hostPath to store some tmp data, the tmp data may be lost.

For Pod don't use emptyDir/hostPath, I think there's no much difference for usage( will involve controller, apiserver, scheduler to do some work as it needs to create a replacement, backoffLimit policy will avoid bad jobSpec & container cause to create too many replacement consume control plane bandwidth).
Might it be a reasonable idea to move the parameter into the Pod? I'd think it could prove valuable to have an upper bound on the restarts on Pods themselves independently of Jobs, and it may solve the "how does the Cronjob controller determine that the limit has been reached" problem more elegantly if the Pod's state automatically transitions to failed once the limit is hit. WDYT?
can anyone tell me if I set the Never and the back off to 3 will it try a restart 3 times? this is very confusing
yes, it will. In fact, setting it to is the only way right now to have a back off limit be applied and effective.
I should mention that my statement is based on Kubernetes 1.8. I don't know if any fixes possibly made it into the upcoming 1.9. Then again, if there were, this issue should have been updated or at least referenced.
this is not working for a cronjob job spec (i am running 1.8.6) in the picture i still get new pods when i kill the previous on (5 created 3 should have been created and the job should have failed) any idea how to get this to actually work? !
what's your spec's ? I don't see it in your manifest.
every minute just for testing
Aren't you risking then that restarting jobs from a failed schedule and those from a succeeding schedule might be overlapping? How does it look like when you set the interval to, say, 10 minutes?
same result with 10 minutes interval
I ran a simplified version of your manifest against my Kubernetes 1.8.2 cluster with the command field replaced by so that every run fails, the schedule set to every 5 minutes, and the back-off limit reduced to 2. Here's my manifest: Initially, I get a series of three failing pods: Then it takes pretty much 5 minutes for the next batch to start up and fail: Interestingly, I only get two new containers instead of three.

The next batch though comes with three again: and so does the final batch I tested: Apart from the supposed off-by-one error, things seem to work for me.
Maybe you can double-check using my manifest?
In the we've stated it should work for both restart policies, I've addressed the problem in It counts pod's restartCount, if overall (iow. all containers in all pods) number exceeds backoff limit the job will fail.
I think the backoffLimit doesn't work at all. My kubernetes version is 1.10.3
See .
NAME READY STATUS RESTARTS AGE job-backofflimit-2-9n74x 0/1 Error 0 34s job-backofflimit-2-bt796 0/1 Error 0 1m job-backofflimit-2-fgktc 0/1 Error 0 49s job-backofflimit-2-gfhlx 0/1 Error 0 1m job-backofflimit-2-gzd54 0/1 ContainerCreating 0 8s job-backofflimit-2-ljlpx 0/1 Error 0 1m job-backofflimit-2-p5fxm 0/1 Error 0 1m job-backofflimit-2-w6s5d 0/1 Error 0 21s apiVersion: batch/v1 kind: Job metadata: name: job-backofflimit-2 spec: backoffLimit: 2 template: spec: containers: name: job-with-python2 image: restartPolicy: Never v1.10.4
9 has the same problem. My cluster is 1.9.6, and i run into the same issue


Question: make -C "${KUBE_ROOT}" WHAT=vendor/k8s.io/code-generator/cmd/import-boss <del> packages=("k8s.io/kubernetes/pkg/..." "k8s.io/kubernetes/cmd/..." "k8s.io/kubernetes/plugin/..." "k8s.io/kubernetes/test/e2e/framework/...") </del> <ins> packages=( "k8s.io/kubernetes/pkg/..." "k8s.io/kubernetes/cmd/..." "k8s.io/kubernetes/plugin/..." "k8s.io/kubernetes/test/e2e/framework/..." "k8s.io/kubernetes/test/integration/..." ) </ins> for d in staging/src/k8s.io/*/; do if [ -d "$d" ]; then packages+=("./vendor/${d#"staging/src/"}...")

What happened: Integration tests import e2e test code today and the dependency makes two drawbacks. It is hard to move test/e2e/framework into staging () Need to run integration tests always even if PRs are just changing e2e test code This PR is for removing this dependency from integration tests. Target dependencies Following files are importing e2e test code from integration tests: [x] () [x] () [x] () and we need to put file for blocking such dependency anymore.
/sig testing /area code-organization
/remove-kind bug
/assign
for /assign


Question: func InitMultiVolumeTestSuite() TestSuite { return &multiVolumeTestSuite{ tsInfo: TestSuiteInfo{ <del> name: "multiVolume", </del> <ins> name: "multiVolume [Slow]", </ins> testPatterns: []testpatterns.TestPattern{ testpatterns.FsVolModePreprovisionedPV, testpatterns.FsVolModeDynamicPV,

created a spread sheet of the slowest running tests for the pull-kubernetes-gce job. cc discussion on slack: spreadsheet: pie chart of test durations from SIGs: ! it seems appropriate to evaluate the state of the tests and reduce these runtimes. possibly move tests to periodic by marking them as [Slow]. /assign /kind design /area test (?)
/remove-kind bug
Are pull jobs the right place to be calculating the time? Or should be be counting in CI instead?
pull is the one timing out, so probably?
Looking at some of the top slowness:: subPath should fail for new directories when readOnly specified in the volumeSource: This test case is no longer defined in master HostPathType: This test case has not been checked in yet and is trying to be in a PR PersistentVolumes-local StatefulSet with pod affinity should use volumes spread across nodes when pod has anti-affinity: This test is already marked as slow in master
Created This should address most of the top slow storage tests


Question: }) } <del> newVM := compute.VirtualMachine{ Location: vm.Location, </del> <ins> newVM := compute.VirtualMachineUpdate{ </ins> VirtualMachineProperties: &compute.VirtualMachineProperties{ HardwareProfile: vm.HardwareProfile, StorageProfile: &compute.StorageProfile{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Instead of using the VM CreateOrUpdate () function of the Azure Go SDK, use the VM Update function () in the azureclient () Why is this needed: Currently, the Azure Cloud Provider is using the VM CreateOrUpdate call when e.g. detach/attaching disks to a VM. In some scenarios, this will re-create a VM that was deleted by an external application. This can lead to orphaned VMs or blocked resources. The Update function will only update an existing VM and will fail if the VM does not exist.
/sig azure
/assign
/sig cloud-provider
Could you describe more about the scenarios? Have you noticed such issues? I wonder adding etags to the API call could help to prevent this?
If you have an application that maintains your K8s clusters, e.g. takes care of OS update/patching. If the application wants to patch the OS of a worker node, it would drain the node first. If draining times out, it will go ahead and delete the VM anyway. But K8s might still be detaching disks. These detach operations, because they use CreateOrUpdate, will recreate the VM. Is there a reason why you would want to create a VM i.e. why CreateOrUpdate is required and Update would not work? AFAIK, Azure Compute RP does not support etags.
that sounds reasonable, thanks change to use function would be safer since it's only detach/attach disk operation. I will do that.
thanks for the information. Reasonable enough. /assign
/milestone v1.16
BTW, the reason why we only use func in the beginning is that Azure Compute go sdk did not provide a func at that time...
Great. Thanks!


Question: t.Errorf("unexpected error: %v", err) } <del> expectedOut := []string{"Unschedulable", "true", "holder"} </del> <ins> expectedOut := []string{"Unschedulable", "true", "holder", `Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource           Requests     Limits --------           --------     ------ cpu                1 (25%)      2 (50%) memory             1Gi (8%)     2Gi (16%) ephemeral-storage  0 (0%)       0 (0%) hugepages-1Gi      0 (0%)       0 (0%) hugepages-2Mi      512Mi (25%)  512Mi (25%)`} </ins> for _, expected := range expectedOut { if !strings.Contains(out, expected) { t.Errorf("expected to find %q in output: %q", expected, out)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: scheduler doesn't assign pod because 'not enough resource: hugepages-1Gi',but the node info show it has enough resource. describe node What you expected to happen: the pod was in normal status like pending and running How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It was change default hugepage size to 1Gi and hugepages=36 numastat was show there has enough hugepage also proc/meminfo was show there has enough resource other info Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
cc do you have thoughts here? I see huge pages and think of you :)
Hmm. Are you sure that there are no other pods using huge page memory ? To me, it looks like the node has (huge page @ 1GiB) memory, you are requsting , but is already used. Can you verify that by listing the pods on the node, and check if some of them are using huge page memory already?
Also, the OS will report that no huge pages are in use as long as no processes have started using the hugepages. Eg. when a pod requesting with is assigned to a node, k8s will then mark the s with huge page memory it as "used". The cgroup controller (if hugetlb cgroup is mounted) will then set the usage limit to . Before the pod actually starts using the huge page memory, the OS will report that no huge page memory is used. When a process inside the pod maps with huge page memory via eg. , the OS will see that the memory is in used. If the process tries to map more than the requested amount, the map will fail (given that the hugetlb cgroup is mounted). To verify if the hugetlb cgroup is mounted, run .

If mounted, it will retrun something like this:
I’m so sorry that there actually has a other pod using hugepagei only see node info available hugepage. so i close the issuethanks a lot
No problem I think this issue is a symptom that something should be done to make it easier to understand. I will look into it.


Question: } } <ins> // TestGarbageCollectorSync ensures that a discovery client error // will not cause the garbage collector to block infinitely. func TestGarbageCollectorSync(t *testing.T) { serverResources := []*metav1.APIResourceList{ { GroupVersion: "v1", APIResources: []metav1.APIResource{ {Name: "pods", Namespaced: true, Kind: "Pod", Verbs: metav1.Verbs{"delete", "list", "watch"}}, }, }, } fakeDiscoveryClient := &fakeServerResources{ PreferredResources: serverResources, Error:              nil, Lock:               sync.Mutex{}, InterfaceUsedCount: 0, } testHandler := &fakeActionHandler{ response: map[string]FakeResponse{ "GET" + "/api/v1/pods": { 200, []byte("{}"), }, }, } srv, clientConfig := testServerAndClientConfig(testHandler.ServeHTTP) defer srv.Close() clientConfig.ContentConfig.NegotiatedSerializer = nil client, err := kubernetes.NewForConfig(clientConfig) if err != nil { t.Fatal(err) } rm := &testRESTMapper{legacyscheme.Registry.RESTMapper()} metaOnlyClientPool := dynamic.NewClientPool(clientConfig, rm, dynamic.LegacyAPIPathResolverFunc) clientPool := dynamic.NewClientPool(clientConfig, rm, dynamic.LegacyAPIPathResolverFunc) podResource := map[schema.GroupVersionResource]struct{}{ {Group: "", Version: "v1", Resource: "pods"}: {}, } sharedInformers := informers.NewSharedInformerFactory(client, 0) alwaysStarted := make(chan struct{}) close(alwaysStarted) gc, err := NewGarbageCollector(metaOnlyClientPool, clientPool, rm, podResource, map[schema.GroupResource]struct{}{}, sharedInformers, alwaysStarted) if err != nil { t.Fatal(err) } stopCh := make(chan struct{}) defer close(stopCh) go gc.Run(1, stopCh) go gc.Sync(fakeDiscoveryClient, 10*time.Millisecond, stopCh) // Wait until the sync discovers the initial resources fmt.Printf("Test output") time.Sleep(1 * time.Second) err = expectSyncNotBlocked(fakeDiscoveryClient) if err != nil { t.Fatalf("Expected garbagecollector.Sync to be running but it is blocked: %v", err) } // Simulate the discovery client returning an error fakeDiscoveryClient.setPreferredResources(nil) fakeDiscoveryClient.setError(fmt.Errorf("Error calling discoveryClient.ServerPreferredResources()")) // Wait until sync discovers the change time.Sleep(1 * time.Second) // Remove the error from being returned and see if the garbage collector sync is still working fakeDiscoveryClient.setPreferredResources(serverResources) fakeDiscoveryClient.setError(nil) err = expectSyncNotBlocked(fakeDiscoveryClient) if err != nil { t.Fatalf("Expected garbagecollector.Sync to still be running but it is blocked: %v", err) } } func expectSyncNotBlocked(fakeDiscoveryClient *fakeServerResources) error { before := fakeDiscoveryClient.getInterfaceUsedCount() t := 1 * time.Second time.Sleep(t) after := fakeDiscoveryClient.getInterfaceUsedCount() if before == after { return fmt.Errorf("discoveryClient.ServerPreferredResources() called %d times over %v", after-before, t) } return nil } </ins> type fakeServerResources struct { PreferredResources []*metav1.APIResourceList Error              error <ins> Lock               sync.Mutex InterfaceUsedCount int </ins> } func (_ *fakeServerResources) ServerResourcesForGroupVersion(groupVersion string) (*metav1.APIResourceList, error) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector.

When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.


Question: RestartPolicy: v1.RestartPolicyAlways, Containers: []v1.Container{ { <del> Name:    "busybox-container", Image:   busyboxImage, </del> <ins> Name:  "busybox-container", Image: busyboxImage, SecurityContext: &v1.SecurityContext{ Capabilities: &v1.Capabilities{ Add: []v1.Capability{"CAP_NET_RAW"}, }, }, </ins> Command: getRestartingContainerCommand("/test-empty-dir-mnt", 0, numRestarts, "echo 'some bytes' >/outside_the_volume.txt; ping -c 1 google.com; echo 'hello world' >> /test-empty-dir-mnt/file;"), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: While trying to run the node conformance tests with CRIO, the test I noticed failures like on the logs of the test containers. This happens because CRIO doesn't ship with capability enabled by default. Since only this test in requires it, we should add this capability only in this test and not runtime wide. Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: CRIO doesn't ship with by default. Anything else we need to know:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: shouldSchedule = false case *predicates.PredicateFailureError: var emitEvent bool <del> // we try to partition predicates into two partitions here: intentional on the part of the operator and not. </del> switch reason { <del> // intentional case predicates.ErrNodeSelectorNotMatch, predicates.ErrPodNotMatchHostName, predicates.ErrNodeLabelPresenceViolated, // this one is probably intentional since it's a workaround for not having // pod hard anti affinity. predicates.ErrPodNotFitsHostPorts, // DaemonSet is expected to respect taints and tolerations predicates.ErrTaintsTolerationsNotMatch: wantToRun, shouldSchedule, shouldContinueRunning = false, false, false // unintentional </del> <ins> // unintentional predicates reasons need to be fired out to event. </ins> case predicates.ErrDiskConflict, predicates.ErrVolumeZoneConflict,

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DaemonSet FailedPlacement Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): OL7.3 Kernel (e.g. ): 4.1.12-94.2.1.el7uek.x86_64 Install tools: Others: What happened: I have a DaemonSet that is only deployed to a subset of nodes in the cluster based on a . I'm seeing events fired with reason for the daemonset pods on nodes that do not fit the selector. What you expected to happen: These events should never fire because they do not meet the node selectors in the pod spec. How to reproduce it (as minimally and precisely as possible): Create a cluster with different sized nodes. E.g. smaller masters than workers. Then deploy the daemonset only to the workers. You will see events triggered for the master nodes that don't meet the requirements. Anything else we need to know:
DaemonSet controller will run predicates to check if node should run daemon pod. In the progress of predicates, it will check if pod fits resources first before other predicates such as PodSelectorMatches. /cc
I think this bug appeared when critical pods were introduced and order of checking predicates changed. The event is useless for daemonsets in current situation :/ I'm not sure if there is a clean way to fix it.
I feel like a clean solution would be to look at the predicate reasons twice. The first time you look at the reasons and keep track if any are not intentional. If all of the reasons are intentional then skip checking the reasons.
actually just changing the order of checks in GeneralPredicates may just work. I'm testing the fix now.
Ok, GeneralPredicates returns all reasons. Error that pod doesn't match the node is also returned but because it's "intentional" there is no event for it.

The event "Node didn't have enough resource" should be sent only when there are no other "intentional" reasons.
Yes, I think maybe we need to check reason list twice: check is in tail of predicate list.
/cc
any PR for this? If not, I can help
I will have some time to work on it next week. If you have a spare time you can create a PR.
/assign


Question: // IsNotMountPoint will be called instead of IsLikelyNotMountPoint. // IsNotMountPoint is more expensive but properly handles bind mounts. func UnmountMountPoint(mountPath string, mounter mount.Interface, extensiveMountPointCheck bool) error { <del> if pathExists, pathErr := PathExists(mountPath); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists { </del> <ins> pathExists, pathErr := PathExists(mountPath) if !pathExists { </ins> glog.Warningf("Warning: Unmount skipped because path does not exist: %v", mountPath) return nil } <del> var notMnt bool var err error if extensiveMountPointCheck { notMnt, err = mount.IsNotMountPoint(mounter, mountPath) } else { notMnt, err = mounter.IsLikelyNotMountPoint(mountPath) </del> <ins> corruptedMnt := isCorruptedMnt(pathErr) if pathErr != nil && !corruptedMnt { return fmt.Errorf("Error checking path: %v", pathErr) </ins> } <ins> return doUnmountMountPoint(mountPath, mounter, extensiveMountPointCheck, corruptedMnt) } </ins> <del> if err != nil { return err } </del> <ins> // doUnmountMountPoint is a common unmount routine that unmounts the given path and // deletes the remaining directory if successful. // if extensiveMountPointCheck is true // IsNotMountPoint will be called instead of IsLikelyNotMountPoint. // IsNotMountPoint is more expensive but properly handles bind mounts. // if corruptedMnt is true, it means that the mountPath is a corrupted mountpoint, Take it as an argument for convenience of testing func doUnmountMountPoint(mountPath string, mounter mount.Interface, extensiveMountPointCheck bool, corruptedMnt bool) error { if !corruptedMnt { var notMnt bool var err error if extensiveMountPointCheck { notMnt, err = mount.IsNotMountPoint(mounter, mountPath) } else { notMnt, err = mounter.IsLikelyNotMountPoint(mountPath) } </ins> <del> if notMnt { glog.Warningf("Warning: %q is not a mountpoint, deleting", mountPath) return os.Remove(mountPath) </del> <ins> if err != nil { return err } if notMnt { glog.Warningf("Warning: %q is not a mountpoint, deleting", mountPath) return os.Remove(mountPath) } </ins> } // Unmount the mount path

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.

What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package workqueue import ( "testing" "time" "k8s.io/client-go/pkg/api/v1" ) func TestNoMemoryLeak(t *testing.T) { timedQueue := NewTimedWorkQueue() timedQueue.Add(&TimedWorkQueueItem{Object: &v1.Pod{}, StartTime: time.Time{}}) item, _ := timedQueue.Get() timedQueue.Add(item) // The item should still be in the timedQueue. timedQueue.Done(item) item, _ = timedQueue.Get() timedQueue.Done(item) if len(timedQueue.Type.processing) != 0 { t.Errorf("expect timedQueue.Type.processing to be empty!") } } </del>

The TimedWorkQueue object appears to lack a critical feature of the original WorkOueue, namely that similar items are aggregated together. By adding the time member to to the key type, it ensures that (time, item) pairs are unique-- you can add the same item multiple times as long as the time differs. This removes the guarantee that two goroutines wouldn't process the same item at the same time.
Iirc, the timed queue feeds a regular queue, so the guarantee still holds
Sorry, I was thinking of the delaying queue, which does key correctly
I'm going to remove timed workqueue. It's used to gather metrics for queues used in the garbage collector. This feature is supported by a regular workqueue now.