Question: } return ti.Before(tj) } <ins> // OverlapsWith returns true when two given deployments are different and overlap with each other func OverlapsWith(current, other *extensions.Deployment) (bool, error) { if current.UID == other.UID { return false, nil } currentSelector, err := unversioned.LabelSelectorAsSelector(current.Spec.Selector) if err != nil { return false, fmt.Errorf("deployment %s/%s has invalid label selector: %v", current.Namespace, current.Name, err) } otherSelector, err := unversioned.LabelSelectorAsSelector(other.Spec.Selector) if err != nil { // Broken selectors from other deployments shouldn't block current deployment. Just log the error and continue. glog.V(2).Infof("Skip overlapping check: deployment %s/%s has invalid label selector: %v", other.Namespace, other.Name, err) return false, nil } return (!currentSelector.Empty() && currentSelector.Matches(labels.Set(other.Spec.Template.Labels))) || (!otherSelector.Empty() && otherSelector.Matches(labels.Set(current.Spec.Template.Labels))), nil } </ins>

A developer accidentally made two deployments with labels that overlapped, such that one of the deployment's selectors matched the pods from both deployments. In the repro below the labels are exactly the same, but this also happened one one set of labels is a subset of another. Expected: An error? Actual: Cluster CPU went past 100% and everything else broke. Crazy thrashing with kubernetes shutting down pods & restarting new ones so quickly that we could not even get logs. They never made it past the Pending state. We are running kubernetes 1.2 on GKE. Here's a minimal repro, just apply it (on a cluster you don't need! D: )
See also , . My guess is that this is due to pod relabeling. cc
this shouldn't be an issue in 1.4 anymore
I just ran into this issue and I'm on v1.4.4. My cluster goes into a crazy Pod creation frenzy even though the number of Pod replicas in both the deployments is set to 1 because one of the overlapping Deployment seems to be "redeploying" itself thousands of times. On a single node k8s it quickly reached the limit of 110 pods and makes the cluster unusable until I delete one of the deployments. This is a very nasty bug as it will overrun the whole cluster. I my case the matching labels are only partial. Both deployments have the label "run=nginx" but one of them has a second selected label as well. What happens is that one of the deployments gets the annotation: "" And that Deployment goes into a loop of "redeploying" itself so it tries to create hundreds of pods and overruns the cluster. Looking at the live Deployment yaml it shows "generation : 6616". I've attached the two test Deployments files to reproduce this and I noticed that the ORDER you deploy them in MATTERS! $ kubectl create -f deployment- (and wait until it is fully deployed) $ kubectl create -f deployment- (and watch all hell break loose) I recommend using a disposable cluster even though, if you are quick enough, you can recover by deleting one of the Deployments (it will take a while!) and then its ReplicaSet.

Client/Server versions: Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Yes it's a bug. deployment-one has selector & label run=my-nginx deployment-two has selector & label run=my-nginx, other-label=somevalue deployment-two was updated later In the code we look for overlapping deployments by checking if one's selector matches the other's label: In this case, deployment-one selector matches deployment-two (but not the other way around), so only deployment-one knows there's an overlapping deployment. The we choose the deployment that's last updated to ignore (skip syncing): In this case, deployment-one assumes deployment-two will skip syncing itself, so deployment-one continues. But deployment-two doesn't know there's an overlapping deployment. Updating the statement for finding overlapping deployment should solve it.
So you mean that the original/older Deployment always "wins" and the newer deployed one doesn't get any Pods launched or the Deployment "relaunched" even though after deployment the older Deployment is the one in violation (like in my use case)?
Sent the
In this case they overlap with each other (I mean, not only the older one is in violation). The one whose selector is last updated should be the one which stops syncing. Otherwise it'll be surprising when someone adds a new deployment or update a deployment's selector causes my existing deployment to stop working. The bug is because the latter one doesn't know it overlap with the former one. With the fix it can detect other overlapping deployments correctly.
Hello Guys! I think that I have a relevant question related to this issue. I would appreciate any answer. That configuration, which generate this bad behavior. I mean to have a common label between two deploys wouldn't be required to implement a canary release?

If the canary release was just a different nginx's version? I am asking this because I have this configuration ,even though I don't have this mess hapening, my deployment never reachs my desired state. If I describe this deployment I got a warning: !!!WARNING!!! This deployment has overlapping label selector with deployment "my-nginx" and won't behave as expected. Please fix it before continue. I am using kubernetes 1.5.1, have I miss something related to concept or implementation of canary release?
can you post the deployment yaml files you are running? Seems like they are overlapping. Are both deployments broken or is it just one of them?
Hello Just one was broken, unfortunately I overwrite the files with another deployment configuration. Probably. I was my mistake, because now everything is working. I got my canary release through the files that follows: I am sharing because I got here ins this thread with this problem.These configurations could be useful to someone with the same problem. Thank you for your attention to my question.

Deployment stable apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ise spec: replicas: 3 template: metadata: labels: app: "ise" track: "stable" spec: containers: name: "nginx" image: "nginx:1.8.1" ports: containerPort: 80 lifecycle: preStop: exec: command: ["/usr/sbin/nginx","-s","quit"] livenessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1 Deployment canary apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ise-canary spec: replicas: 1 template: metadata: labels: app: "ise" track: "canary" spec: containers: name: "nginx" image: "nginx:1.9.12" ports: containerPort: 80 lifecycle: preStop: exec: command: ["/usr/sbin/nginx","-s","quit"] livenessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1 Service apiVersion: v1 kind: Service metadata: name: ise spec: ports: port: 80 selector: app: ise type: LoadBalancer
Between two overlapping deployments, one being broken is expected today - you should also see an event about it. This won't be an issue in releases after 1.6. I am assuming deployments in old clusters (1.6+) in the future will contain owner references plus maybe we will have selector generation. Related discussion in and , Adriano S. Fonseca < :


Question: legacyregistry.MustRegister(watchCacheCapacityDecreaseTotal) legacyregistry.MustRegister(WatchCacheCapacity) legacyregistry.MustRegister(WatchCacheInitializations) <ins> legacyregistry.MustRegister(WatchCacheReadWait) </ins> }) }

Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // remove old KMS provider <ins> // verifyIfKMSTransformersSwapped sometimes passes even before the changes in the encryption config file are observed. // this causes the metrics tests to fail, which validate two config changes. // this may happen when an existing KMS provider is already running (e.g., new-kms-provider-for-secrets in this case). // to ensure that the changes are observed, we added one more provider (kms-provider-to-encrypt-all) and are validating it in verifyIfKMSTransformersSwapped. </ins> encryptionConfigWithoutOldProvider := ` kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: t.Errorf("got am %v, expected access mode of ReadOnlyMany", ams[0]) } } <ins> func TestTranslateInTreePVToCSIVolIDFmt(t *testing.T) { g := NewGCEPersistentDiskCSITranslator() pdName := "pd-name" tests := []struct { desc               string topologyLabelKey   string topologyLabelValue string wantVolId          string }{ { desc:               "beta topology key zonal", topologyLabelKey:   v1.LabelFailureDomainBetaZone, topologyLabelValue: "us-east1-a", wantVolId:          "projects/UNSPECIFIED/zones/us-east1-a/disks/pd-name", }, { desc:               "v1 topology key zonal", topologyLabelKey:   v1.LabelTopologyZone, topologyLabelValue: "us-east1-a", wantVolId:          "projects/UNSPECIFIED/zones/us-east1-a/disks/pd-name", }, { desc:               "beta topology key regional", topologyLabelKey:   v1.LabelFailureDomainBetaZone, topologyLabelValue: "us-central1-a__us-central1-c", wantVolId:          "projects/UNSPECIFIED/regions/us-central1/disks/pd-name", }, { desc:               "v1 topology key regional", topologyLabelKey:   v1.LabelTopologyZone, topologyLabelValue: "us-central1-a__us-central1-c", wantVolId:          "projects/UNSPECIFIED/regions/us-central1/disks/pd-name", }, } for _, tc := range tests { t.Run(tc.desc, func(t *testing.T) { translatedPV, err := g.TranslateInTreePVToCSI(&v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{tc.topologyLabelKey: tc.topologyLabelValue}, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: pdName, }, }, }, }) if err != nil { t.Errorf("got error translating in-tree PV to CSI: %v", err) } if got := translatedPV.Spec.PersistentVolumeSource.CSI.VolumeHandle; got != tc.wantVolId { t.Errorf("got translated volume handle: %q, want %q", got, tc.wantVolId) } }) } } </ins>

Was just perusing the code and looks like we translate regional disks into the "zonal volume ID fmt", and there doesn't seem to be any tests for regional disks running so we should confirm that this works and fix it if it doesn't /assign /cc
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /assign
/assign


Question: // Handles returns true if the object count evaluator needs to track this attributes. func (o *objectCountEvaluator) Handles(a admission.Attributes) bool { operation := a.GetOperation() <del> return operation == admission.Create || (o.allowCreateOnUpdate && operation == admission.Update) </del> <ins> return operation == admission.Create </ins> } // Matches returns true if the evaluator matches the specified quota with the provided input item

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.


Question: Resource:     fmt.Sprintf("spn:%s", apiserverID), Type:         tokenType, }, <ins> environment: environment, </ins> clientID:    clientID, tenantID:    tenantID, apiserverID: apiserverID,

What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: NodeName:                 node.Name, } cleanup, err := utils.CreateFromManifests(config.Framework, driverNamespace, func(item interface{}) error { <del> if err := utils.PatchCSIDeployment(config.Framework, o, item); err != nil { return err } // Remove csi-external-health-monitor-agent and // csi-external-health-monitor-controller // containers. They are not needed for any of the // tests and in practice apparently caused enough // overhead that even unrelated tests timed out. For // example, in the pull-kubernetes-e2e-kind test, 43 // out of 5771 tests failed, including tests from // sig-node, sig-cli, sig-api-machinery, sig-network. switch item := item.(type) { case *appsv1.StatefulSet: var containers []v1.Container for _, container := range item.Spec.Template.Spec.Containers { switch container.Name { case "csi-external-health-monitor-agent", "csi-external-health-monitor-controller": // Remove these containers. default: // Keep the others. containers = append(containers, container) } } item.Spec.Template.Spec.Containers = containers } return nil </del> <ins> return utils.PatchCSIDeployment(config.Framework, o, item) </ins> }, h.manifests...) if err != nil {

newer hostpath deployments but removed the csi-external-health-monitor-agent because it seemed to cause overhead without being needed for the tests. We should re-enable it and check whether that really causes tests to fail because of the extra load.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /area test


Question: case PodTerminated: // Older API versions did not contain enough info to map to PodSucceeded *out = newer.PodFailed <ins> case PodSucceeded: *out = newer.PodSucceeded </ins> case PodUnknown: *out = newer.PodUnknown default:

This happened to me several times while debugging an e2e test. My branch was based on commit and had changes only to
docker inspect says the container exited 0
kubelet HTTP says the pod exited 0:
kubectl says it failed:
Events does not show any success or failure even. Seems like it doesn't show that for any pods, succeeded or failed.
Apiserver sees the exit code 0:
The problem might be related to there being zero values for startedAt or finishedAt. The command may have exited so quickly that docker could not record these values, and we may be somehow treating unset values for these as failure.
adding sleep 5 to my command causes docker to set the startedAt and finishedAt to reasonable values, but the pod still shows up as failed. Note that I use a different pod name each time I try to avoid that sort of obvious mistake.
So, the apiserver v1beta1 message knows that it is terminated with exit code 0. But, both kubectl and my go code in my go e2e test thing this means failed. I'm starting to suspect the conversion code.
The problem is that a v1beta1 pod can never see that a pod is failed or succeded. So it defaults to PodFailed. Grrrrrrraaaaahhhhhh!
In line 184 of
This is going to be confusing for people unless we change this.
I believe the latest fix for PodStatus should resolve the issue here. I will double confirm it before closing it.

We should do this after 1.7. xref /sig testing cc
Is this the task that needs completed - Add support to Ubuntu e2e test-infra
That is different, this is about running GPU tests as part of PR pre-submit tests. Right now we skip them because they are "[Serial]".
I can work on this issue in conjunction with . Is this issue just a matter of adding GPU tests to a different suite or are there more subtasks?
Anymore progress on this?
Thanks for the reminder. I just submitted an internal request for quota to run these tests. Once that is approved, I will add a PR in test-infra to run GPU tests on all k/k PRs. It will be non-blocking initially and then become blocking later on.
Awesome, thanks!


Question: "time" "github.com/google/go-cmp/cmp" <ins> "github.com/stretchr/testify/require" </ins> "sigs.k8s.io/yaml" appsv1 "k8s.io/api/apps/v1"

Sending a patch built from a typed client to SSA will force a write to etcd (and bump resourceVersion) since creationTimestamp is serialized to null. This is a follow-up of which doesn't solve the issue properly. Sending a no-op SSA request with creationTimestamp to null shouldn't do anything Kubectl apply an object with creationTimestamp null twice and see that it updates the object. Probably something like this would cause it:
/sig api-machinery /wg api-expression /triage accepted /assign /priority important-soon


Question: // BetaAnnotationExternalTraffic is the beta version of AlphaAnnotationExternalTraffic. BetaAnnotationExternalTraffic = "service.beta.kubernetes.io/external-traffic" ) <del> // NeedsHealthCheck Check service for health check annotations func NeedsHealthCheck(service *v1.Service) bool { // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationExternalTraffic, BetaAnnotationExternalTraffic} { if l, ok := service.Annotations[annotation]; ok { if l == AnnotationValueExternalTrafficLocal { return true } else if l == AnnotationValueExternalTrafficGlobal { return false } else { glog.Errorf("Invalid value for annotation %v: %v", annotation, l) } } } return false } // GetServiceHealthCheckNodePort Return health check node port annotation for service, if one exists func GetServiceHealthCheckNodePort(service *v1.Service) int32 { if !NeedsHealthCheck(service) { return 0 } // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationHealthCheckNodePort, BetaAnnotationHealthCheckNodePort} { if l, ok := service.Annotations[annotation]; ok { p, err := strconv.Atoi(l) if err != nil { glog.Errorf("Failed to parse annotation %v: %v", annotation, err) continue } return int32(p) } } return 0 } // GetServiceHealthCheckPathPort Return the path and nodePort programmed into the Cloud LB Health Check func GetServiceHealthCheckPathPort(service *v1.Service) (string, int32) { if !NeedsHealthCheck(service) { return "", 0 } port := GetServiceHealthCheckNodePort(service) if port == 0 { return "", 0 } return "/healthz", port } </del>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS m4.2xlarge OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-36-generic Install tools: kubeadm Others: 1 master and 1 minion What happened: This states that if using kube-proxy will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. However when I annotate my NodePort type service with this annotation I see the following error in the logs of kube-proxy of the node where the only pod of the service is running: After this message the service stops working outside of the cluster on the specified NodePort. What you expected to happen: I expected to see that only the with the pod on it serves traffic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Sorry, I have found the issue, I had a setup where the none of the node names were the same as the hostname of the kube-proxy pod and this caused the issue, where none of the Pods running on a hosts were recognised as local pods. I have fixed this issue by manually setting the hostname with in kube-proxy. However I still insist that this error message is misleading and shouldn't be printed in case of NodePort type services, but only for just LoadBalancers.
I met the same problem in the aws cloud providers, by adding - the hostname-override to solve this problem. good


Question: } if deleted { klog.InfoS("Removed real server from graceful delete real server list", "realServer", name) <del> q.remove(rs) </del> <ins> delete(q.list, rs.String()) </ins> } } return success

! no panic Kubernetes version (use ): k8s 1.16
/sig Network
Could you fix this? I think the function needs a lock while being executed.
/triage accepted /good-first-issue todo: add a lock around the body of this function: Similar to
/assign
Oh I see, thank you for letting me know, I'll keep that in mind going forward. :)
Anyone actually on this? It seems like a good fix to have for 23 and maybe backport
I'll have a try.
There is a reproducer here
Oops, I just peeked at the time of the pull request and not realized that you've got such deep conversation.
/remove-good-first-issue /unassign
/assign
hi ! someone work on this now? I would like to try it.
Yes, working on it currently.
are you still planning to give this a shot?
Is there no one working on this? If so, I'd submit a PR for it this week. :)


Question: initialConfig.ContainerLogMaxSize = testContainerLogMaxSize }) <del> ginkgo.It("should be rotated and limited to a fixed amount of files", func(ctx context.Context) { </del> <ins> var logRotationPod *v1.Pod ginkgo.BeforeEach(func() { </ins> ginkgo.By("create log container") pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{

Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial] DevicePlugin [Serial] [Disruptive] Can schedule a pod that requires a device It appears a bit flaky on testgrid, but more recently been failing more often No response example - No response /sig node
/cc
/triage accepted /assign /priority important-soon
I ran this test locally 15 times back to back and did not run into this failure. The test passed locally. I suspect this to be a case where pod cleanup is not happening properly. In this test, we are relying on the number of PodResources and based on the error it looks like we have an addtional pod running other than the two expected pods (device plugin pod and test pod requesting device) but can't be 100% sure because we don't have enough logs to support that claim. Going to add more logs to expose output obtained from the podresource API endpoint.


Question: klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewNoNewPrivsAdmitHandler(klet.containerRuntime)) if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) { <del> klet.nodeLeaseController = nodelease.NewController(klet.clock, klet.heartbeatClient, string(klet.nodeName), kubeCfg.NodeLeaseDurationSeconds, klet.onRepeatedHeartbeatFailure) </del> <ins> klet.nodeLeaseController = nodelease.NewController(klet.clock, klet.heartbeatClient, string(klet.nodeName), kubeCfg.NodeLeaseDurationSeconds, kubeCfg.NodeStatusUpdateFrequency.Duration, klet.onRepeatedHeartbeatFailure) </ins> } klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewProcMountAdmitHandler(klet.containerRuntime))

What happened: The node lease feature becomes beta and enables by default in v1.14. After upgrading our cluster to v1.14, we found some nodes become not ready occasionally and later become ready again after a few seconds. In order to have a quick failure detection and recovery, our kubelet has a flag setting and controller-manager has another flag setting . If we use node lease feature, the kubelet doesn't report it's status as frequently as specified by flag , and use node lease mechanism to report that node is alive. the default interval at which node lease renew is hardcoded 10s, but it can't be exactly 10s and may be a little bit more than 10s because of some network latency. when controller-manager doesn't receive messages from kubelet in every 10s in our case, it will mark this node as unreachable and not ready state, but in the next reconcile cycle, it receives the lease update message and mark this node as reachable again, so the node status is flapping between ready and not ready state. This seems a breaking change that setting kubelet's lease renewing interval to a hardcoded 10s, even though our node status update frequency has a little bit small value, node lease should align kubelet's previous behavior and renew as frequently as to reporting alive message, otherwise controller-manager will mark this node as unreachable. What you expected to happen: So we should not merely set node lease renewing interval to 10s, maybe depending on the has a more heuristic value. How to reproduce it (as minimally and precisely as possible): a v1.14 kubernetes cluster set flag to kubelet and to controller-manager. wait a few minute, 'll find some log regarding in controller-manager's log. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /kind bug /assign
smells like a regression WDYT?
cc
- not a regression - I think the only issue is that we may be missing some documentation, but other than that I think everything looks fine.

So how it works now is: There are two fields in Kubelet Config: (a) NodeStatusUpdateFrequency: (b) NodeStatusReportFrequency The comments there are pretty detailed and you should read them, but in short the first one roughly translates to how frequenty the lease object is send and the second how frequenty node status is updated (assuming nothing changed in the meantime). IIRC, there was a decision to not expose it via a flag, because flags in kubelet are deprecated and we're moving to config. So you can actually mitigate the problem by setting the second field in the config I mentioned above to some smaller value (like 5s) and then you will get Node Status updates every 5s as you want. So it's possible to achieve what you want without any code changes. That said, enabling for changing the default lease duration to something larger than 10s, sounds reasonable.
How about making the default lease duration 12s (or 14s) ? I can send out a PR once a new default is agreed upon.
thanks for comments. But what if we both want node lease feature enables (as this is an indeed helpful feature) and heartbeat with apiserver within some smaller interval (like 5s) via node lease mechanism?
IMHO, we should add a filed in Kubelet Config to let user tune the node lease renewing interval, WDYT?
That sounds like the cleanest option to me (rather than some playing with existing params). - who is the owner of the feature on Node side.


Question: // First return value indicates whether a matching pod exists on a node that matches the topology key, // while the second return value indicates whether a matching pod exists anywhere. // TODO: Do we really need any pod matching, or all pods matching? I think the latter. <del> func (c *PodAffinityChecker) anyPodMatchesPodAffinityTerm(pod *v1.Pod, allPods []*v1.Pod, node *v1.Node, term *v1.PodAffinityTerm) (bool, bool, error) { </del> <ins> func (c *PodAffinityChecker) anyPodMatchesPodAffinityTerm(pod *v1.Pod, pods []*v1.Pod, nodeInfo *schedulercache.NodeInfo, term *v1.PodAffinityTerm) (bool, bool, error) { </ins> if len(term.TopologyKey) == 0 { return false, false, fmt.Errorf("empty topologyKey is not allowed except for PreferredDuringScheduling pod anti-affinity") }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --It is not a secret that . In my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. However, in most practical scenarios users do not need all the flexibility. For example, people often do not use arbitrary labels in topology key of the affinity rules. They often use node, zone, or region as the topology. Node in particular is used often. That's why I am proposing a fast path for the case that topology key is Node. Today, the affinity/anti-affinity predicate does not have any distinction for topology keys and in the cluster regardless. The logic should recognize "" and scan only pods on the same node for this topology key. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature /sig scheduling cc/
ref/
/assign
GitHub didn't allow me to assign the following users: misterikkit. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
We found some other simple optimizations that had a significant impact on the affinity predicate. I will upload PRs for those as well.
Reopening this as the performance of anti-affinity still doesn't seem to be satisfactory enough on large clusters (i.e 5k nodes) - and it's causing them to fail to startup. Ref: issue I can help with evaluating any performance improvements if needed.
The change sent improved performance of the predicate, but apparently it is not enough to make our large scale scalability tests happy.
This issue is not about general performance improvment of affinity/anti-affinity. It was to implement a particular improvment which is already implemented. There are other performance improvement issues. I am going to close this one.
Sounds good. Thanks.
How come with kops 1.10.0 I still see all kube-dns pods on the same node because the kube-dns deployment doesn't specify any anti-affinity rule?
I think you are correct. you can double-check with the option to . I have a v1.9 cluster in front of me, and kube-dns has no affinity rules. (I didn't use kops though)


Question: // Find all sibling pods in the service and post to their /write handler. func contactOthers(state *State) { <ins> var ( versionInfo *version.Info err         error ) </ins> sleepTime := 5 * time.Second // In large cluster getting all endpoints is pretty expensive. // Thus, we will limit ourselves to send on average at most 10 such

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: [] [sig-node] PreStop should call prestop when killing a pod [Conformance] Since when has it been failing: N/A Testgrid link: N/A Reason for failure: When spawning a Pod, it's container might start before all its networking plumbing has been completed. Thus, if the container tests / needs its network connectivity, it mgiht fail. The ` image tries to connect to the Kubernetes API, but if it cannot, the Pod will end up in an Error state. Retrying the Kubernetes API connection will avoid this issue. Anything else we need to know:
/sig node
/remove-sig node /sig testing


Question: var _ = utils.SIGDescribe("CSI mock volume", func() { type testParameters struct { <del> attachable      bool </del> <ins> disableAttach   bool </ins> attachLimit     int registerDriver  bool podInfoVersion  *string

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear


Question: endpoint:             "/var/log/kubelet/plugins/myplugin/csi.sock", versions:             []string{"0.2.0", "v1.2.3"}, foundInDeprecatedDir: true, <del> shouldFail:           true, </del> <ins> shouldFail:           false, </ins> }, { pluginName:           "test.plugin",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: CSI allows 0.x drivers only in the deprecated kubelet plugin dir. CSI allows 1.x drivers only in the new kubelet plugin dir. If a driver implements both 0.x and 1.x (which is unlikely since the side car containers do not support this), kubelet requires the driver live in new kubelet plugin dir. So CSI drivers that implement both 0.x and 1.x if they want a seamless transition between k8s 1.12 to 1.13 would need to deploy in both old and new dir. What you expected to happen: argues that this is adding churn for CSI authors. He recommends allowing any driver that implements 0.x (even if it implements 1.x) to use either new or old (not just new). /milestone v1.13 If release team approves, PR is small and I can get prepare it pretty quickly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
discussed in the burndown meeting, please go ahead and prepare the PR
/sig storage /priority critical-urgent /milestone v1.13
we need to wait for a fix for this before lifting code freeze


Question: } else if vm.InstanceView != nil && vm.InstanceView.PlatformFaultDomain != nil { // Availability zone is not used for the node, falling back to fault domain. failureDomain = strconv.Itoa(int(*vm.InstanceView.PlatformFaultDomain)) <ins> } else { err = fmt.Errorf("failed to get zone info") klog.Errorf("GetZoneByNodeName: got unexpected error %v", err) ss.deleteCacheForNode(name) return cloudprovider.Zone{}, err </ins> } return cloudprovider.Zone{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The cluster is release-1.16. When a VMSS is manually scaled up, some pointers in of the newly created nodes are nil. Here's the list of the missing ones: PlatformUpdateDomain PlatformFaultDomain - - - The strikeouts mean they are missing but we don't focus on them. What you expected to happen: These pointers are not supposed to be nil. How to reproduce it (as minimally and precisely as possible): a cluster with VMSS, k8s version release-1.16 and ccm enabled. up the VMSS. Anything else we need to know?: Note that this issue only happens when ccm is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /area provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /area provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider /area provider/azure /milestone v1.17


Question: arm=arm32v6/alpine:3.8 arm64=arm64v8/alpine:3.8 ppc64le=ppc64le/alpine:3.8 <ins> s390x=s390x/alpine:3.8 </ins>

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: tc.runTest(t) } <ins> func TestScaleDownIgnoresFailedPods(t *testing.T) { tc := testCase{ minReplicas:          2, maxReplicas:          6, initialReplicas:      5, desiredReplicas:      3, CPUTarget:            50, CPUCurrent:           28, verifyCPUCurrent:     true, reportedLevels:       []uint64{100, 300, 500, 250, 250}, reportedCPURequests:  []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, useMetricsAPI:        true, reportedPodReadiness: []v1.ConditionStatus{v1.ConditionTrue, v1.ConditionTrue, v1.ConditionTrue, v1.ConditionTrue, v1.ConditionTrue, v1.ConditionFalse, v1.ConditionFalse}, reportedPodPhase:     []v1.PodPhase{v1.PodRunning, v1.PodRunning, v1.PodRunning, v1.PodRunning, v1.PodRunning, v1.PodFailed, v1.PodFailed}, } tc.runTest(t) } </ins> func TestTolerance(t *testing.T) { tc := testCase{ minReplicas:         1,

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: set +o nounset set +o errexit } <ins> run_kubectl_diff_same_names() { set -o nounset set -o errexit create_and_use_new_namespace kube::log::status "Test kubectl diff with multiple resources with the same name" output_message=$(KUBECTL_EXTERNAL_DIFF=find kubectl diff -Rf hack/testdata/diff/) kube::test::if_has_string "${output_message}" 'v1.Pod:.*.test' kube::test::if_has_string "${output_message}" 'apps.v1.Deployment:.*.test' kube::test::if_has_string "${output_message}" 'v1.ConfigMap:.*.test' kube::test::if_has_string "${output_message}" 'v1.Secret:.*.test' set +o nounset set +o errexit } </ins>

What happened: I get a diff, when I specify a file directly, but on recursive mode, no diff is shown anymore: What you expected to happen: Recursive mode should work How to reproduce it (as minimally and precisely as possible): Have a bunch of YAML manifest, but with name collisions, e.g. Service , Deployment , etc Anything else we need to know?: I think the problem is that all objects have the same name, which should not be a problem per se. Only heapster-binding has a different name, and looking at the generated temp files, only and are present: Environment: Kubernetes version (use ): $ kubectl version --short Client Version: v1.13.0 Server Version: v1.13.0 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
I think the files are overwriting each other?
Yeah, good catch, I'll try to fix that asap
Thanks! By the way, very nice work with dry-run and diff. I was waiting for this.
Feel free to look at the PR and give your opinion!


Question: ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a LoadBalancer service on the same nodes", func(ctx context.Context) { // requires cloud load-balancer support <del> e2eskipper.SkipUnlessProviderIs("gce", "gke", "aws", "azure") </del> <ins> e2eskipper.SkipUnlessProviderIs("gce", "gke", "azure") </ins> ns := f.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(ctx, cs, 1) framework.ExpectNoError(err)

When running the e2e tests on AWS, the following two tests are failing: with the stated cause being: these tests were with Expected the tests to pass Run the loadbalancer test suite on AWS While investigating the root cause of these failures, i cam across this function in the cloud-provider-aws: This function checks to ensure that a specific annotation has been applied to the Service object to determine if it is an NLB. The test in question creates the Service object with no annotations placed upon it, this in turn causes the test to fail. This confirms the need for the user to add the annotation. It is a continuation of a discussion that started in the SIG Cloud Provider meeting from 15 Feb 2023. <detailsi don't have a live cluster for this anymore, but it was version 1.26.1 </details<detailsAWS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cloud-provider
when exploring this issue, i noticed we have this function in a different load balancer test: that function appears to have some specific logic for adding/removing annotations depending on the cloud provider. i thought this might be a place we could also use for AWs and the new tests, but i'm not sure if it is appropriate.
the test should be as much agnostic as possible, theoretically we tried to remove all the checks for cloud providers but Adding annotations and supporting all permutations on each cloud provider will be hard to support in the long term. If AWS doesn't support it we just can skip them in AWS
/assign
thanks , agreed on all points
please go ahead and tag me


Question: "fmt" "net" "strings" <ins> "sync" </ins> "syscall" "github.com/vishvananda/netlink"

I gathered e2e test results of suite for last 100 runs. The following is the list of flaky tests. Some might not be related to . I will start with the service tests. Any help would be appreciated. Detail list: Jenkins project:
cc/ Cluster team is working on to make kubenet default. Once we switch this to production, we don't need to restart docker to configure cbr0.
After a bunch of PRs got merged, the kubenet suite is much more stable. One more known flake can be fixed by I will post new flake tests here.
Just bump into a new failure: One of the nodes () got panic: Looks like it crashed at shaper reset.: Can you take a look?


Question: return nodeMem } <del> // getTotalAllocatableMemory gets the sum of all agent node's allocatable memory func getTotalAllocatableMemory(f *framework.Framework) *resource.Quantity { selector := labels.Set{"kubernetes.io/os": "windows"}.AsSelector() nodeList, err := f.ClientSet.CoreV1().Nodes().List(context.TODO(), metav1.ListOptions{ LabelSelector: selector.String(), }) framework.ExpectNoError(err) ginkgo.By("Summing allocatable memory across all agent nodes") totalAllocatable := resource.NewQuantity(0, resource.BinarySI) for _, node := range nodeList.Items { status := node.Status totalAllocatable.Add(status.Allocatable[v1.ResourceMemory]) } return totalAllocatable } </del> // modified from https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/kubelet/config.go#L110 // the proxy version was causing and non proxy used a value that isn't set by e2e func decodeConfigz(contentsBytes []byte) (*kubeletconfig.KubeletConfiguration, error) {

The test : is failing because it assumes the memory is the same across all nodes. A change in the configuration changed that assumption and the test started to fail. the test to assign the correct amount of memory to each of the nodes to fill them up. create a Windows cluster with 3 nodes. Two have 6gb of allocatable memory, 1 has 14gb of allocatable memory. Run the test above and it will fail because it tries to create 3 pods with 9 gb of memory. The first pod will land on the node with 14gb of memory but the other two pods won't be able to find a node with enough memory (9gb pod 6 gb nodes). fixes this in the configuration /sig windows <details</details<details</details<details</details<details</details<details</details<details</details
/triage accepted /priority important-long-term
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted /priority important-long-term Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please inform that Am I reasonable here or not. We need to update the in all the cases inside the (, and ) similar to what was done in PR, but now we need to update it in all cases. If you could please confirm that I am reasonable here then I can begin to make these changes in a PR.
sorry I could have been clearer in the issue. The test it's self needs to be updated to handle nodes that do not have consistent memory. The logic in the test is wrong if the nodes don't all have the same memory values: This results in pods that won't fit on some of the nodes and leaves the pods pending. The test should handle this situation by making sure pods filled up memory but can be scheduled across the nodes. Let me know of need more clarification.
So, here we have to fix the logic because the logic is wrong for the case when the nodes don't have the same memory values.

Is the node memory value is equal to which gets assigned to variable in ?My understanding of tackling the issue : So, we need to change because it is the total memory and when it is fixed then the pods present will get appropriate memory limit by . So, if updating the logic for the value of is our focus, such that the it (memory limit) is changed according to the nodes, how can we achieve that ? Am I thinking reasonable here? If my understanding is correct till now then how we will be changing the code for assigning the memory value for a node such that each of the nodes get relevant amount of memory based on there need to work properly.
Yes, I think think the simplest thing to so would be get a list of windows nodes iterate though each one, looking up the allocatable memory for the node, creating a pod to fill that memory and assign it to that node for scheduling via , ensuring the scheduling was successful. Then once the nodes are all full deploy the pod that should go into pending (code should stay the same). pseudo code:
Do we have some examples in the repo where we do something similar for other nodes ? First we need to make function or is it already present ? Confirming : Also Do we need to run the whole code(After Modifying) inside in loop for every windows node and if even one return fails, the test fails.
I need some guidance. Could you please take a look on previous comment.
example of getting the nodes and memory total: no, basically need to replace some of the code in that function with the sudo code in
Why are we not deploying both the pods of in one loop only (if we want memory of pending pods memory same as nodes memory). Also after we deploys both the pods, do we will be running another loop similar to for checking the failure. I am not much sure about my changes, can I make a PR so that I can discuss about my code and make appropriate changes further ?
Sounds good
Does functions , and already exists or we have to make them because I am not able to find, have you used these names as short forms for other functions which do same as their names ?


Question: endpoint: %s`, listener.Addr().String())), os.FileMode(0755)); err != nil { t.Fatal(err) } <ins> testAPIServerTracing(t, listener, []string{ "--tracing-config-file=" + tracingConfigFile.Name(), "--egress-selector-config-file=" + egressSelectorConfigFile.Name(), }, ) } func testAPIServerTracing(t *testing.T, listener net.Listener, apiserverArgs []string) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.APIServerTracing, true)() traceFound := make(chan struct{}) defer close(traceFound) srv := grpc.NewServer() traceservice.RegisterTraceServiceServer(srv, &traceServer{ traceFound: traceFound, filterFunc: containsNodeListSpan}) go srv.Serve(listener) defer srv.Stop() </ins> // Start the API Server with our tracing configuration testServer := kubeapiservertesting.StartTestServerOrDie(t, kubeapiservertesting.NewDefaultTestServerOptions(), <del> []string{"--tracing-config-file=" + tracingConfigFile.Name()}, </del> <ins> apiserverArgs, </ins> framework.SharedEtcd(), ) defer testServer.TearDownFn()

While trying to enable API Server Tracing while using k3s, the API Server component crashes. The only way to avoid this crash is to disable egress selectors in k3s entirely. I originally opened an issue in the k3s repo (see ), and we determined that the issue seems to be an assumption that a dialer will always be returned if there's no error, but in this case there's no grpc egress configured so is returning and causing a panic at API Server Tracing to function properly. Please see the linked k3s issue for a repro case; However, I believe the minimal setup for this would be to configure cluster egress without defining a grpc egress selector and then attempting to enable API server tracing. panic stack trace -- contents of egress selector config file in direct mode- contents of egress selector config in 'agent' mode - contents of api server tracing config file -- <details25.2+k3s1 </details<detailsDigital Ocean Droplet </details<details</details<details</details<details</details<details</details
/sig instrumentation
/assign /triage accepted Thanks


Question: // This implementation is shared between Linux and NsEnterMounter func doCleanSubPaths(mounter Interface, podDir string, volumeName string) error { <del> glog.V(4).Infof("Cleaning up subpath mounts for %s", podDir) </del> // scan /var/lib/kubelet/pods/<uid>/volume-subpaths/<volume>/* subPathDir := filepath.Join(podDir, containerSubPathDirectoryName, volumeName) <ins> glog.V(4).Infof("Cleaning up subpath mounts for %s", subPathDir) </ins> containerDirs, err := ioutil.ReadDir(subPathDir) if err != nil { if os.IsNotExist(err) {

Fix status: [x] fixed in master in [x] fixed in release-1.10 branch in , will be in 1.10.0 [x] fixed in release-1.9 branch in , will be in 1.9.7 [x] fixed in release-1.8 branch in , will be in 1.8.11 [x] fixed in release-1.7 branch in , will be in 1.7.16 In SubPath fixes, PR adds bind mount for subpaths. It uses as the path name. The volumename in this path is the container's mount volume name which is also called OuterVolumeSpecName in . This becomes an issue for volume reconstruction when kubelet restarts. Volume scans the volume directories and recover the InnerVolumeSpecName (which is used to construct pod volume paths). Volume reconstruction has not enough information to recover the OuterVolumeSpecName and figure out which volume it maps to. In case kubelet needs to clean up mounts (it happens when volume spec cannot be reconstructed), it will fail to clean up the subpaths because of lacking the OuterVolumeSpecName. We think it is better to use InnerVolumeSpecName for volume subPath instead of OuterVolumeSpecName so that volume clean up can work. The issue we might have is that some users might already deploy a cluster with subpath might already . If the users perform an in-place upgrade, since new version will use InnerVolumeSpecName which might have conflicts with some other older version's OuterVolumeSpecName among different volumes. This change requires clear documentation and communication with the users.
cc
This impacts reconstruction of PVCs, where the inner and outer names are different.
/assign
While testing the e2es against a cluster without the fix, I noticed that subsequent pods using volumes failed to start. My theory is that the unmount volume failed because the subpath was still mounted, and that blocked the volume manager from processing other pods. But I thought that volume manager is supposed to run the unmount volume operation in a separate thread per volume, so this shouldn't happen. Need to inspect the code more carefully to confirm the theory. could the volume reconstruction refactor have a unmount volume call in the main volume manager thread?
runs in reconciler's goroutine - so if that function is blocked, it will not be able to process any more volumes.

Was the problem that failed to unmount volume but got hung/stuck ?
I haven't confirmed for sure, but my guess is that unmount volume failed because the subpath was still mounted
moving into the milestone for discussion /milestone v1.10
/priority critical-urgent /status in-progress
/status approved-for-milestone per discussion in sig-release meeting today
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Re-opening to track fixes in release branches
keeping in the v1.10 milestone until merges
since this is now fixed in 1.10, I'm taking it out of the milestone so that the release team doesn't keep checking it.
Can this issue be closed now?


Question: ChunkSize:          cmdutil.DefaultChunkSize, }, } <del> o.drainer.OnPodDeletedOrEvicted = o.onPodDeletedOrEvicted </del> <ins> o.drainer.OnPodDeletionOrEvictionFinished = o.onPodDeletionOrEvictionFinished o.drainer.OnPodDeletionOrEvictionStarted = o.onPodDeletionOrEvictionStarted </ins> return o } <del> // onPodDeletedOrEvicted is called by drain.Helper, when the pod has been deleted or evicted func (o *DrainCmdOptions) onPodDeletedOrEvicted(pod *corev1.Pod, usingEviction bool) { </del> <ins> // onPodDeletionOrEvictionFinished is called by drain.Helper, when eviction/deletetion of the pod is finished func (o *DrainCmdOptions) onPodDeletionOrEvictionFinished(pod *corev1.Pod, usingEviction bool, err error) { </ins> var verbStr string if usingEviction { <del> verbStr = "evicted" </del> <ins> if err != nil { verbStr = "eviction failed" } else { verbStr = "evicted" } } else { if err != nil { verbStr = "deletion failed" } else { verbStr = "deleted" } } printObj, err := o.ToPrinter(verbStr) if err != nil { fmt.Fprintf(o.ErrOut, "error building printer: %vn", err) fmt.Fprintf(o.Out, "pod %s/%s %sn", pod.Namespace, pod.Name, verbStr) } else { _ = printObj(pod, o.Out) } } // onPodDeletionOrEvictionStarted is called by drain.Helper, when eviction/deletion of the pod is started func (o *DrainCmdOptions) onPodDeletionOrEvictionStarted(pod *corev1.Pod, usingEviction bool) { if !klog.V(2).Enabled() { return } var verbStr string if usingEviction { verbStr = "eviction started" </ins> } else { <del> verbStr = "deleted" </del> <ins> verbStr = "deletion started" </ins> } printObj, err := o.ToPrinter(verbStr) if err != nil { fmt.Fprintf(o.ErrOut, "error building printer: %vn", err) fmt.Fprintf(o.Out, "pod %s/%s %sn", pod.Namespace, pod.Name, verbStr) } else { <del> printObj(pod, o.Out) </del> <ins> _ = printObj(pod, o.Out) </ins> } }

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: <ins> # This Dockerfile will build an image that is configured # to run Fluentd with an Elasticsearch plug-in and the # provided configuration file. # TODO(satnam6502): Use a lighter base image, e.g. some form of busybox. # The image acts as an executable for the binary /usr/sbin/td-agent. # Note that fluentd is run with root permssion to allow access to # log files with root only access under /var/lib/docker/containers/* # Please see http://docs.fluentd.org/articles/install-by-deb for more # information about installing fluentd using deb package. FROM ubuntu:14.04 MAINTAINER Satnam Singh "satnam@google.com" # Ensure there are enough file descriptors for running Fluentd. RUN ulimit -n 65536 # Install prerequisites. RUN apt-get update &&  apt-get install -y curl &&  apt-get install -y -q libcurl4-openssl-dev make &&  apt-get clean # Install Fluentd. RUN /usr/bin/curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh # Change the default user and group to root. # Needed to allow access to /var/log/docker/... files. RUN sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent # Install the Elasticsearch Fluentd plug-in. RUN /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch # Copy the Fluentd configuration file. COPY td-agent.conf /etc/td-agent/td-agent.conf # Copy a script that determines the name of the host machine # and then patch the Fluentd configuration files and then # run Fluentd in the foreground. ADD run.sh /run.sh # Always run the this setup script. ENTRYPOINT ["/run.sh"] </ins>

After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.


Question: }, { "ImportPath": "github.com/docker/spdystream", <del> "Rev": "29e1da2890f60336f98d0b3bf28b05070aa2ee4d" </del> <ins> "Rev": "e9bf9912b85eec0ed6aaf317808a0eab25e3ca43" </ins> }, { "ImportPath": "github.com/elazarl/go-bindata-assetfs",

Hit this while :
cc/ Andy can you take a look at this failure?
looking now
Working with on a fix
Proposed fix:


Question: v1.ResourceMemory: resource.MustParse("15Mi"), }, Limits: v1.ResourceList{ <del> v1.ResourceMemory: resource.MustParse("15Mi"), </del> <ins> v1.ResourceMemory: resource.MustParse("35Mi"), </ins> }, }

E2eNode Suite: [It] [sig-node] [NodeConformance] Containers Lifecycle when a pod is terminating because its liveness probe fails should execute readiness probe while in preStop, but not liveness [NodeConformance] Has been flaking sinc Testcase addition in It has been constantly failing in our internal CI on /sig node
^^
this is strange, I don't understand why the hook is never executed... is it a matter of waiting more for this to happen? What's the performance of these ppc runners?
On our ppc node it seems to a OOM error. I see the container related to testcase being killed with OOM message. request you to point me out if there is any memory requirement to run the e2e-node suite. Also, is there anything related to this particular test that consumes more memory?
Interesting... not that I knew. These are small containers using BusyBox images, I don't think we can do much smaller than that. Do you have memory pressure on the node? Because we don't set any resource limit on our Pods. Maybe we should try to reduce the env variable?
I shall try reducing the as you suggested. But, the test fails with OOM messages even when run alone (not as part of the complete suite)
Any clue on why we are experiencing OOM error only for this testcase? I am trying to compare pod spec to see if this is defined differently.
I don't think we have anything special in the Pod spec... I might try to replicate exactly one test that works.
cc
I think this is still failing. /reopen
Reopened this issue. <detailsIn response to : I think this is still failing. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
A short preStop hook period looks like an issue. I'll post a PR.
/assign /triage accepted
can we close this?
Yes, I think so as there is no failure recently. /close
Closing this issue. <detailsIn response to : can we close this? /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.ExpectNoError(err, "Failed to patch ephemeral containers in pod %q", format.Pod(pod)) ginkgo.By("checking pod container endpoints") <del> _, err = framework.LookForStringInPodExecToContainer(pod.Namespace, pod.Name, ecName, []string{"/bin/echo", "marco"}, "marco", time.Minute) framework.ExpectNoError(err, "Failed to exec in pod %q ephemeral container %q", format.Pod(pod), ecName) _, err = framework.LookForStringInLog(pod.Namespace, pod.Name, ecName, "polo", time.Minute) framework.ExpectNoError(err, "Failed to find logs in pod %q ephemeral container %q", format.Pod(pod), ecName) </del> <ins> // Can't use anything depending on kubectl here because it's not available in the node test environment output := f.ExecCommandInContainer(pod.Name, ecName, "/bin/echo", "marco") gomega.Expect(output).To(gomega.ContainSubstring("marco")) log, err := e2epod.GetPodLogs(f.ClientSet, pod.Namespace, pod.Name, ecName) framework.ExpectNoError(err, "Failed to get logs for pod %q ephemeral container %q", format.Pod(pod), ecName) gomega.Expect(log).To(gomega.ContainSubstring("polo")) </ins> }) })

E2eNode Suite.[sig-node] Ephemeral Containers will start an ephemeral container in an existing pod node-kubelet-orphans Unknown, possibly was never green No response KEP: /sig node
Also need to fish this test out of the orphans by adding the label
can you please take a look?
yup, happy to /assign
could you explain this more? This label should exist even though the feature is no longer alpha?
Looks like the tests are still passing in the presubmits It looks like the kubectl invocation is asking for a username?
Moved to node features and now the error message is: kubectl isn't available to the node tests? Hmm..
/triage accepted
/reopen The test is flaking now, can you please take a look?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ah, I think I know what the problem is. Didn't get a chance to look at it today but will get to it soon.
FYI, I spot checked several of the recent failures and so far none of them include the fix commit. Will keep an eye on it.
Yes, thank you. Looks very green now


Question: "//pkg/util/mount:go_default_library", "//pkg/volume:go_default_library", "//pkg/volume/testing:go_default_library", <ins> "//vendor/github.com/Azure/go-autorest/autorest/to:go_default_library", </ins> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/types:go_default_library",

If I look at the code of azureFile plugin, it has fsGroup as an argument, but don't see its usage anywhere I also don't see anywhere explicitely stated that azureFile volume supports fsGroup.
/sig storage
For those landing here, a separate path to get the mount point to have the right permissions is to add this annotation to the PV metadata: This makes the plugin mount the volume in the Pod under those UID,GID. Obviously, this implies that one knows what those are for their workload in the container ...
If i create a PV like this, still mounted with the default root used into POD. StorageClass PV
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc Let's check whether works for azure file.
/sig azure
is another way to set groupid(gid) While according to this , we also need to support Here is the code about setting fsgroup from securityContext. I will enable this feature and do a test, it should be 3 lines of code. /assign
BTW, this is all for Linux(UserID, GroupID), is there an identical setting for Windows container?
No, setting user for volumes doesn't work on Windows. Only the rw/ro options are implemented.
According to my investigation, though I put the following code in azure file , it does not take effect. The reason is azure file(cifs) mount does not support gid setting after file system mount, by default would be 0(root) So in summary, set fsGroup code in azure file is useless, user could set gid by mountOptions in storage class, here is an Let me know if you hav any concern about this issue Update: Root cause is azure file(cifs) mount does not support gid setting after file system mount, while azure disk supports.
is this true, that cifs does not support guid setting?
Of course cifs support gid setting, see example: I mean cifs does not support second time gid setting, kind of differnet than azure disk.

Anyway, I am working on a PR to support if no gid setting in azure file storage class, while if both sets, only gid from storage class would be used.
while the code is quite different than just add below 3 lines of code, actually below does not work: I will show you the PR tomorrow.


Question: wg.Wait() } <ins> type fakeTimeBudget struct{} func (f *fakeTimeBudget) takeAvailable() time.Duration { return 2 * time.Second } func (f *fakeTimeBudget) returnUnused(_ time.Duration) {} </ins> func TestDispatchEventWillNotBeBlockedByTimedOutWatcher(t *testing.T) { backingStorage := &dummyStorage{} cacher, _, err := newTestCacher(backingStorage)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: } } <ins> if proxier.network.networkType == "Overlay" { providerAddress := proxier.network.findRemoteSubnetProviderAddress(ep.ip) isNodeIP := (ep.ip == providerAddress) isPublicIP := (len(providerAddress) == 0) klog.Infof("Endpoint %s on overlay network %s is classified as NodeIp: %v, Public Ip: %v", ep.ip, hnsNetworkName, isNodeIP, isPublicIP) containsNodeIP = containsNodeIP || isNodeIP containsPublicIP = containsPublicIP || isPublicIP } </ins> // Save the hnsId for reference LogJson(newHnsEndpoint, "Hns Endpoint resource", 1) hnsEndpoints = append(hnsEndpoints, *newHnsEndpoint)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Overlay mode, When 2 service has the same backend node IP, the LB was not programmed correctly causing datapath issues. What you expected to happen: Service data path to work properly How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /sig windows These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: $(error VERSION is undefined) endif cp -r ./* ${TEMP_DIR} <del> mkdir -p ${TEMP_DIR}/cni-bin </del> cp ../../../${HYPERKUBE_BIN} ${TEMP_DIR} chmod a+rx ${TEMP_DIR}/hyperkube <del> cd ${TEMP_DIR} && sed -i.back "s|ARCH|${QEMUARCH}|g" Dockerfile </del> cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile <del> ifeq ($(CACHEBUST),1) cd ${TEMP_DIR} && sed -i.back "s|CACHEBUST|$(shell uuidgen)|g" Dockerfile endif ifeq ($(ARCH),$(HOSTARCH)) # When building "normally", remove the whole line, it has no part in the image cd ${TEMP_DIR} && ${SED_CMD} "/CROSS_BUILD_/d" Dockerfile else cd ${TEMP_DIR} && ${SED_CMD} "s/CROSS_BUILD_//g" Dockerfile # When cross-building, only the placeholder "CROSS_BUILD_" should be removed # Register /usr/bin/qemu-ARCH-static as the handler for ARM binaries in the kernel docker run --rm --privileged multiarch/qemu-user-static:register --reset curl -sSL --retry 5 https://github.com/multiarch/qemu-user-static/releases/download/$(QEMUVERSION)/x86_64_qemu-${QEMUARCH}-static.tar.gz | tar -xz -C ${TEMP_DIR} endif # Download CNI curl -sSL --retry 5 https://storage.googleapis.com/kubernetes-release/network-plugins/cni-${ARCH}-${CNI_RELEASE}.tar.gz | tar -xz -C ${TEMP_DIR}/cni-bin </del> docker build --pull -t ${REGISTRY}/hyperkube-${ARCH}:${VERSION} ${TEMP_DIR} rm -rf "${TEMP_DIR}"

I'm pretty sure that building the hyperkube images takes longer than building kubernetes itself now. At least a large part of this seems to be that we download and install ~172 extra debian packages several times (once for each arch). <details</detailsNot sure how to improve this - maybe better caching? Do we actually want to install these extra packages? Should we use a better base image than ? cc
cc (not really sure who "owns" hyperkube, if anyone)
The hyperkube binary is one thing. I think the image ends up being big as we drag in so much stuff. I wasn't involved in creating the docker image so I can't comment. Ideally this thing would be slim and would have minimal external dependencies. Fixing this starts with mapping why we have each of those deps imported.
I actually have written a such patch in my local env, but I never uploaded. It removed the docker and docker-multinode-specific stuff and all the unused deps. Also, we will come one step closer once is in. Should I submit the patch so you can take a look at it?
sure!
This needs to be triaged as a release-blocker or not for 1.5
punting to v1.6.
The main problem here is that we're installing huge packages for supporting some volume types: git nfs-common glusterfs-client cifs-utils With containerized volume mounters we'll get rid of that.
When it comes to local rebuilding, should help here. As I've already noted in the PR, maybe we should make CACHEBUST=0 the default for all hack/dev- scripts. This would at least ease development and testing.
can this be closed? or moved to 1.7?
should this be in v1.7? is there an appropriate sig to own this?
cries nope, not v1.7. I'm still not sure who owns hyperkube.
Everyone and no one I suppose. Yeah, agree it's sad.
/sig release guess we're saying sig-release owns hyperkube under the general guise of "cutting bits" until/unless there is a better home for it
Yeah, hyperkube definitely belongs to sig-release as it hasn't been owned by a sig so far...


Question: expectedCondition:            "ScaleDownLimit", }, { <ins> name:                         "scaleDown with percent policy and previous scale up and down events", scaleUpEvents:                generateEventsUniformDistribution([]int{2}, 120), scaleDownEvents:              generateEventsUniformDistribution([]int{4}, 120), specMinReplicas:              1, specMaxReplicas:              1000, scaleDownRules:               generateScalingRules(0, 0, 50, 180, 0), currentReplicas:              10, prenormalizedDesiredReplicas: 1, expectedReplicas:             6, expectedCondition:            "ScaleDownLimit", }, { </ins> // corner case for calculating the scaleDownLimit, when we changed pod or percent policy after a lot of scaleDown events // in this case we shouldn't allow scale down, though, the naive formula will suggest that scaleDownlimit is more then CurrentReplicas (100+30-10% > 100) name:                         "scaleDown with previous events preventing further scale down",

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale