Question: "printers.go", ], deps = [ <del> "//cmd/kubeadm/app/apis/kubeadm:go_default_library", </del> "//federation/apis/federation:go_default_library", "//federation/apis/federation/v1beta1:go_default_library", "//federation/client/clientset_generated/federation_clientset:go_default_library",

/kind bug What happened: doesn't show role Workaround: What you expected to happen: According to and How to reproduce it (as minimally and precisely as possible): I just deployed a multi master kubernetes environment with kops: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS Install tools: kops, terraform
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli
I don't think this is a bug. The node role label is not used by kubeadm anymore. Related codes have been changed in by
Hi Thanks for the quick reply I'm actually using the as you can see in the workaround above. I'm not using kubeadm role label.
Your favorite topic came up again. Hi I do think this is a bug. As said, kubectl shouldn't show anything based on the old kubeadm master label, but it should probably on the label, but absolutely on the label. cc as you have opinions on this as well cc


Question: c.versionSupported = supported if !supported { <del> return fmt.Errorf("version %s not supported", verToStr(ver)) </del> <ins> return fmt.Errorf( "CSI Driver does not support version %s. Instead it supports versions %s", verToStr(ver), versToStr(vers)) </ins> } glog.V(4).Info(log("version %s supported", verToStr(ver)))

/kind bug What happened: When a CSI driver reports a version of the CSI spec which does not match that stored in Kubernetes, Kubernetes prints the following message: What you expected to happen: Message should be: How to reproduce it (as minimally and precisely as possible): Use a CSI driver that reports version 0.0.0. Then deploy an application pod with a PVC from this driver. Application pod description will show the actual message above. Environment: Kubernetes version (use ): 1.9.1
/sig storage
/assign lpabon
hi, I can help fix that.
I did not see your PR until after I submitted.


Question: // queueActionLocked appends to the delta list for the object. // Caller must lock first. func (f *DeltaFIFO) queueActionLocked(actionType DeltaType, obj interface{}) error { <ins> return f.queueActionInternalLocked(actionType, actionType, obj) } // queueActionInternalLocked appends to the delta list for the object. // The actionType is emitted and must honor emitDeltaTypeReplaced. // The internalActionType is only used within this function and must // ignore emitDeltaTypeReplaced. // Caller must lock first. func (f *DeltaFIFO) queueActionInternalLocked(actionType, internalActionType DeltaType, obj interface{}) error { </ins> id, err := f.KeyOf(obj) if err != nil { return KeyError{obj, err} } // Every object comes through this code path once, so this is a good <del> // place to call the transform func.  If obj is a // DeletedFinalStateUnknown tombstone, then the containted inner object // will already have gone through the transformer, but we document that // this can happen. In cases involving Replace(), such an object can // come through multiple times. </del> <ins> // place to call the transform func. // // If obj is a DeletedFinalStateUnknown tombstone or the action is a Sync, // then the object have already gone through the transformer. // // If the objects already present in the cache are passed to Replace(), // the transformer must be idempotent to avoid re-mutating them, // or coordinate with all readers from the cache to avoid data races. // Default informers do not pass existing objects to Replace. </ins> if f.transformer != nil { <del> var err error obj, err = f.transformer(obj) if err != nil { return err </del> <ins> _, isTombstone := obj.(DeletedFinalStateUnknown) if !isTombstone && internalActionType != Sync { var err error obj, err = f.transformer(obj) if err != nil { return err } </ins> } }

We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used.

FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted


Question: lb.OnUpdate([]api.Endpoints{ {JSONBase: api.JSONBase{ID: "echo"}, Endpoints: []string{net.JoinHostPort("127.0.0.1", port)}}}) <del> p := NewProxier(lb) </del> <ins> p := NewProxier(lb, "127.0.0.1") </ins> proxyPort, err := p.addServiceOnUnusedPort("echo") if err != nil {

kube-proxy listens on all ports, but should be configurable. create a flag to specify which interface to listen on. Thanks for finding this bug.
For context, some providers apparently do external IPs by having a whole interface that gets 1:1 NAT without any firewalling. We do not want service ports on those interfaces. , Brian Ketelsen wrote:


Question: } else if found { t.Errorf("unexpected metadata.unknown=%#v: expected this to be pruned", unknown) } <ins> if generation, found, err := unstructured.NestedInt64(obj.UnstructuredContent(), "metadata", "generation"); err != nil { t.Errorf("unexpected error getting metadata.generation: %v", err) } else if !found { t.Errorf("expected metadata.generation=1: got: %d", generation) } else if generation != 1 { t.Errorf("unexpected metadata.generation=%d: expected this to be set to 1", generation) } </ins> } func TestInvalidObjectMetaInStorage(t *testing.T) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: return } <del> // enforce a timeout of at most requestTimeoutUpperBound (34s) or less if the user-provided // timeout inside the parent context is lower than requestTimeoutUpperBound. ctx, cancel := context.WithTimeout(ctx, requestTimeoutUpperBound) defer cancel() </del> <ins> // DELETECOLLECTION can be a lengthy operation, // we should not impose any 34s timeout here. // NOTE: This is similar to LIST which does not enforce a 34s timeout. </ins> ctx = request.WithNamespace(ctx, namespace) outputMediaType, _, err := negotiation.NegotiateOutputMediaType(req, scope.Serializer, scope)

In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?

The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign


Question: } clientRuleAnnotation := fmt.Sprintf("%s=%s", NLBClientRuleDescription, lbName) healthRuleAnnotation := fmt.Sprintf("%s=%s", NLBHealthCheckRuleDescription, lbName) <del> vpcCIDRs, err := c.getVpcCidrBlocks() if err != nil { return err } </del> for sgID, sg := range clusterSGs { sgPerms := NewIPPermissionSet(sg.IpPermissions...).Ungroup() if desiredSGIDs.Has(sgID) { <del> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", healthCheckPorts, vpcCIDRs); err != nil { </del> <ins> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", healthCheckPorts, subnetCIDRs); err != nil { </ins> return err } if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, clientProtocol, clientPorts, clientCIDRs); err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws


Question: return errors.Wrap(err, "unable to fetch CoreDNS current installed version and ConfigMap.") } <del> var hasCoreDNSMigrationFailed bool if IsCoreDNSConfigMapMigrationRequired(corefile) { </del> <ins> canMigrateCorefile, err := isCoreDNSVersionSupported(client) if err != nil { return err } corefileMigrationRequired, err := isCoreDNSConfigMapMigrationRequired(corefile, currentInstalledCoreDNSVersion) if err != nil { return err } if !canMigrateCorefile { klog.Warningf("the CoreDNS Configuration will not be migrated due to unsupported version of CoreDNS. " + "The existing CoreDNS Corefile configuration and deployment has been retained.") } if corefileMigrationRequired && canMigrateCorefile { </ins> if err := migrateCoreDNSCorefile(client, coreDNSConfigMap, corefile, currentInstalledCoreDNSVersion); err != nil { // Errors in Corefile Migration is verified during preflight checks. This part will be executed when a user has chosen // to ignore preflight check errors. <del> hasCoreDNSMigrationFailed = true </del> <ins> canMigrateCorefile = false </ins> klog.Warningf("the CoreDNS Configuration was not migrated: %v. The existing CoreDNS Corefile configuration has been retained.", err) if err := apiclient.CreateOrRetainConfigMap(client, coreDNSConfigMap, kubeadmconstants.CoreDNSConfigMap); err != nil { return err

What happened: Coredns failed to start in 1.17 deployment when it tried to retain a backed up Corefile with error message: Caddyfile via flag: open /etc/coredns/Corefile: no such file or directory What you expected to happen: Coredns starts normally with the backed-up Corefile How to reproduce it (as minimally and precisely as possible): Deploy a k8s 1.17 cluster and trigger coredns Corefile migration and make it fail. In our case, this was triggered in a three master -setup with 'old' method of running 'kubeadm init' for each master. Also the core-dns docker image used was tagged with '1.6.7-1', apparently causing the corefile migration version check to fail Anything else we need to know?: This seems to be caused by Both 'key' and 'path' are set to 'Corefile-backup' when this function is called. Should the 'path' always stay as 'Corefile' because that is where the container expects to find the config file? Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:51:57Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:47:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Custom deployment in Openstack environment OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" Kernel (e.g. ): 12.14-197.29-default Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
/assign
/area kubeadm


Question: case desiredReplicas > scaleUpLimit: setCondition(hpa, autoscalingv2.ScalingLimited, v1.ConditionTrue, "ScaleUpLimit", "the desired replica count is increasing faster than the maximum scale rate") desiredReplicas = scaleUpLimit <ins> // Ensure that even if the scaleUpLimit is greater // than the maximum number of replicas, we only // set the max number of replicas as desired. if scaleUpLimit > hpa.Spec.MaxReplicas { setCondition(hpa, autoscalingv2.ScalingLimited, v1.ConditionTrue, "TooManyReplicas", "the desired replica count was more than the maximum replica count") desiredReplicas = hpa.Spec.MaxReplicas } </ins> case hpa.Spec.MinReplicas != nil && desiredReplicas < *hpa.Spec.MinReplicas: // make sure we aren't below our minimum var statusMsg string

/kind bug What happened: Since upgrading to Kube 1.7.6, I'm seeing my HPAs scale above it's maxReplicas. Because of this, it's rapidly creating pods, then terminating them, and repeat. What you expected to happen: I expected the HPA to never exceed maxReplicas. How to reproduce it (as minimally and precisely as possible): The HPA is logging: This seems to happen with the deployment's is low, and the HPA shows the deployment using more than 175% of it's If I increase the deployment's to better match actual usage (so that the deployment is using 120% or less), it occurs much less often. I deliberately set the low, to ensure services were over provisioned during the first roll out of HPA. I think the switch in after should be converted to individual if() statements. Anything else we need to know?: I am only setting on the deployment, not Looking at the code in , the version that worked (1.6.2) would only scale up and down by 1. The 1.7.6+ code is attempting to scale up by a larger amount. The new code seems appropriate in large clusters, where maxReplicas is in the tens or hundreds. It doesn't seem appropriate for clusters where maxReplicas is in the single digits. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T19:06:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-14T06:36:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: AWS m4.2xlarge, 5 nodes OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kops Version 1.7.0 Others:
There are no sig labels on this issue.

Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Yeah, I think the bug is that we have a case short-circuiting another case.
I think your right - we are not checking whether . I submitted a diff with a fix and regression test.


Question: return result } <del> func (t *timeBudget) returnUnused(unused time.Duration) { </del> <ins> func (t *timeBudgetImpl) returnUnused(unused time.Duration) { </ins> t.Lock() defer t.Unlock() if unused < 0 {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: mv "${TMP_LICENSE_FILE}" "${dest_dir}/LICENSE" done <ins> # copy licenses for forked code from vendor and third_party directories (cd "${KUBE_ROOT}" &&  find vendor third_party -iname 'licen[sc]e*' -o -iname 'notice*' -o -iname 'copying*' |  grep -E 'third_party|forked' |  xargs tar -czf - | tar -C "${TMP_LICENSES_DIR}" -xzf -) </ins> # Leave things like OWNERS alone. rm -f "${LICENSES_DIR}/LICENSE" rm -rf "${LICENSES_DIR}/vendor" <ins> rm -rf "${LICENSES_DIR}/third_party" </ins> mv "${TMP_LICENSES_DIR}"/* "${LICENSES_DIR}"

we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


Question: }), }, { <del> name:          "invalid-values", </del> <ins> name:          "invalid-multiple-match-expressions-values", </ins> expectSuccess: false, pv: testVolumeWithNodeAffinity(t, &v1.VolumeNodeAffinity{ Required: &v1.NodeSelector{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: The terms are ANDed, which is not correct:
cc


Question: func DoCleanupNode(client clientset.Interface, nodeName string, strategy PrepareNodeStrategy) error { var err error for attempt := 0; attempt < retries; attempt++ { <del> node, err := client.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) </del> <ins> var node *v1.Node node, err = client.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) </ins> if err != nil { return fmt.Errorf("Skipping cleanup of Node: failed to get Node %v: %v", nodeName, err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The value of err in was always nil, and it will lead to incorrect behavior in some case. Assign to in Use functions in any place. (In all environments) /assign /sig scheduling /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "os" <del> "k8s.io/kubernetes/cmd/kubectl/app" </del> <ins> "k8s.io/kubernetes/pkg/kubectl/cmd" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" </ins> ) func NewKubectlServer() *Server { <ins> cmd := cmd.NewKubectlCommand(cmdutil.NewFactory(nil), os.Stdin, os.Stdout, os.Stderr) localFlags := cmd.LocalFlags() localFlags.SetInterspersed(false) </ins> return &Server{ name:        "kubectl", SimpleUsage: "Kubernetes command line client", Long:        "Kubernetes command line client", Run: func(s *Server, args []string) error { <del> os.Args = os.Args[1:] if err := app.Run(); err != nil { os.Exit(1) } os.Exit(0) return nil </del> <ins> cmd.SetArgs(args) return cmd.Execute() </ins> }, <ins> flags: localFlags, </ins> } }

When trying to use kubectl from hyperkube I'm met with "unknown flag" errors. It looks as if the flags aren't being passed through to kubectl but hyperkube is trying to handle them itself (and subsequently giving an error when you give it flags it doesn't understand). You can reproduce simply by running: And you'll see the error:
I have a fix up in PR .


Question: CPUUtilization: &extensions.CPUTargetUtilization{TargetPercentage: cpu}, }, } <del> _, errHPA := rc.framework.Client.Extensions().HorizontalPodAutoscalers(rc.framework.Namespace.Name).Create(hpa) </del> <ins> var errHPA error if useV1 { _, errHPA = rc.framework.Client.Autoscaling().HorizontalPodAutoscalers(rc.framework.Namespace.Name).Create(hpa) } else { _, errHPA = rc.framework.Client.Extensions().HorizontalPodAutoscalers(rc.framework.Namespace.Name).Create(hpa) } </ins> expectNoError(errHPA) }

lightweight test which can be run in gce suite test which verifies that version v1 works


Question: state state.State } <del> func (m *fakeManager) Start(activePods ActivePodsFunc, podStatusProvider status.PodStatusProvider, containerRuntime runtimeService) { </del> <ins> func (m *fakeManager) Start(activePods ActivePodsFunc, sourcesReady config.SourcesReady, podStatusProvider status.PodStatusProvider, containerRuntime runtimeService) { </ins> klog.Info("[fake cpumanager] Start()") }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: return original, nil } <ins> // discardNullValuesFromPatch discards all null property values from patch. // It traverses all slices and map types. func discardNullValuesFromPatch(patchV interface{}) { switch patchV := patchV.(type) { case map[string]interface{}: for k, v := range patchV { if v == nil { delete(patchV, k) } else { discardNullValuesFromPatch(v) } } case []interface{}: for _, v := range patchV { discardNullValuesFromPatch(v) } } } </ins> // mergeMapHandler handles how to merge `patchV` whose key is `key` with `original` respecting // fieldPatchStrategy and mergeOptions. func mergeMapHandler(original, patch interface{}, schema LookupPatchMeta,

Note: I'm using OpenShift. s/oc/kubectl/ below if that's how you roll. I'm using a command similar to the following to idempotently ensure a label is absent on a Namespace object: If the object's contains anything other than , this works fine: the resulting object has no key. However, if is initially absent, this command adds it with an entry with the empty string as a value: Running the command again removes the entry and the . Using this logic in a controller causes a hot loop as we thrash back and forth between the above and no labels. Using a value in a dict in a StrategicMerge patch should idempotently ensure the key is absent, even if the dict is initially absent/empty. I didn't try this on anything other than a Namespace; and I didn't try it on any dict other than . Workaround: Add any other label, as seen above. <details</details<detailsAWS </details<details</details<details</detailsI believe the issue is on the server, but... v0.9.3-0.- </details<details</details
/sig api-machinery
/assign /cc /triage accepted
can you look into it?
It looks that this idempotency problem is fixed in latest releases(after being label as you mentioned). All responses are legit; Kubernetes Version
The bug happens when there are no other labels. Are you saying the "fix" was for kubernetes to always add a label to the object?
That was an incidental change specific to namespaces. I'd expect the issue still exists for other types
you are both right. This is incidental "fix" for namespaces. I'll look at others to find the actual problem.


Question: } } <ins> // updateContainerResourceAllocation updates AllocatedResources values // (for cpu & memory) from checkpoint store func (kl *Kubelet) updateContainerResourceAllocation(pod *v1.Pod) { for _, c := range pod.Spec.Containers { allocatedResources, found := kl.statusManager.GetContainerResourceAllocation(string(pod.UID), c.Name) if c.Resources.Requests != nil && found { if _, ok := allocatedResources[v1.ResourceCPU]; ok { c.Resources.Requests[v1.ResourceCPU] = allocatedResources[v1.ResourceCPU] } if _, ok := allocatedResources[v1.ResourceMemory]; ok { c.Resources.Requests[v1.ResourceMemory] = allocatedResources[v1.ResourceMemory] } } } } </ins> // HandlePodUpdates is the callback in the SyncHandler interface for pods // being updated from a config source. func (kl *Kubelet) HandlePodUpdates(pods []*v1.Pod) {

After enabling In-Place Pod Vertical Scaling, if deploy a pod without setting memory (or cpu) request, kubelet will be failed at its second restart kubelet restart successfully kubelet with In-Place Pod Vertical Scaling enabled a pod without setting memory request, e.g: kubelet for the first time kubelet for the second time will fail with error: After deploying the pod, the following information is saved in the file : Then restart kubelet for the first time, the relevant content in the file becomes: When deploying the pod, as there are no records in the checkpoint (), kubelet will get resource config from directly and save it to the checkpoint (without memory request) When restarting kubelet for the first time, it will restore the previously saved data from the checkpoint. Since the memory request was not previously set, it will be set to an empty value and resaved to the checkpoint When restarting kubelet for the second time, it will process as a value of 0 (not empty), which resulting in the inconsistency between the calculated checksum during data recovery and the previous one <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
cc
/triage accepted
/assign
xlink: /cc
thanks for finding this issue and the fix. It looks good with suggestion. For beta, I wonder if we can do away with local checkpoint and rely on PodStatus as persistent store and source of truth for allocated resources. It will simplify the implementation.
Using resource.Quantity in checkoutPoint is a bad usage. I found except the case in this issue, whenever adding a static pod to the kubelet, then restart the kubelet, restore the resize status checkpoint will fail, casing kubelet panic. The reason is , before restart kubelet, the checkpoint struct in kubelet will looks like: and the the podstatusmanager_state file is looks like: and after restart kubelet and recover the checkoutpoint state, the checkpoint struct in kubelet will looks like: the difference is between "kube-scheduler:map[cpu:{i:{value:6 scale:-1} d:{Dec:<nil} s: Format:DecimalSI}" and "kube-scheduler:map[cpu:{i:{value:600 scale:-3} d:{Dec:<nil} s:600m Format:DecimalSI} ".

It's caused by json Marshal and Unmarshal of resource.Quantity. So, I think Using resource.Quantity in checkoutPoint is a bad case. We should consider using PodStatus in apiserver instead of checkoutPoint.
Is this problem solved? Now I can still reproduce this problem in the official version of 1.28. Thanks~
Could anyone please let me know what is the state of the art to solve this issue?
we have seen the similar problem after enabling InPlaceUpdate pod resources and could we reopen this ?
can we just do marshal&unmarshal before storing state checkpoint to avoid the issue of resource.Quantity?
I am +1 on doing away with the checkpointing code altogether and leveraging PodStatus. We discussed this in the Kubernetes Contributor Summit in Chicago. I just have not been able to find the cycles to work on it :(


Question: ], "@io_bazel_rules_go//go/platform:linux": [ "//pkg/util/procfs:go_default_library", <del> "//staging/src/k8s.io/api/rbac/v1:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library",

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: }, }, { <ins> Name: "vol02", VolumeSource: v1.VolumeSource{ RBD: &v1.RBDVolumeSource{}, }, }, { </ins> Name: "vol3", VolumeSource: v1.VolumeSource{ AzureDisk: &v1.AzureDiskVolumeSource{}, }, }, <ins> { Name: "vol03", VolumeSource: v1.VolumeSource{ AzureDisk: &v1.AzureDiskVolumeSource{}, }, }, </ins> }, }, }

Currently I describe a Pod that is in and fails to attach/mounts its volume: Note that the generated events are shown in 4 section. The only difference between the messages is how the are ordered. I suggest to sort the unmounted/unattached) volumes for the event message. This will allow to group all events and properly have the age and the occurrences count reported. I would expect the unmounted/unattached) volumes to be sorted in the events' message and to do not have several group of events for the same issue. See above. No response <details</details<detailsAWS but does not matter for this issue. </details<details</details<details</details<details</details<details</details
/sig storage
/assign qingwave
/triage accepted


Question: // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) <del> if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn(podBackend2)); err != nil { </del> <ins> if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn(podBackend2, podClient)); err != nil { </ins> logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: // IsMissingVersionError func (v *versionConverter) IsMissingVersionError(err error) bool { <del> return runtime.IsNotRegisteredError(err) </del> <ins> return runtime.IsNotRegisteredError(err) || isNoCorrespondingTypeError(err) </ins> }

Found by failing TestApplyConvertsManagedFieldsVersion test in () It seems that conversion depends on schema info that is collected from served API versions only. However, managedFields stores data in etcd from REST versions served in previous releases which may no longer be served. That means that we have to be able to deal with managedFields persisted via API calls to versions that are no longer served, but which we still have and understand conversions for. /cc /sig api-machinery /wg apply /priority important-soon
This may be fixed in a more recent version of structured-merge-diff. I'll look into it
/assign
At one point we were using the types registered to the scheme to generate the openAPI definitions used by apply, but we switched to using the installed endpoints instead, because it was less invasive (We needed to expose which types are convertible to which other types, and also a way to skip generating openAPI for internal types). We might want to revisit the first approach, given this use case.


Question: } } <ins> // TestWaitForWithEarlyClosingWaitFunc tests WaitFor when the WaitFunc closes its channel. The WaitFor should // always return ErrWaitTimeout. func TestWaitForWithEarlyClosingWaitFunc(t *testing.T) { stopCh := make(chan struct{}) defer close(stopCh) start := time.Now() err := WaitFor(func(done <-chan struct{}) <-chan struct{} { c := make(chan struct{}) close(c) return c }, func() (bool, error) { return false, nil }, stopCh) duration := time.Now().Sub(start) // The WaitFor should return immediately, so the duration is close to 0s. if duration >= ForeverTestTimeout/2 { t.Errorf("expected short timeout duration") } if err != ErrWaitTimeout { t.Errorf("expected ErrWaitTimeout from WaitFunc") } } // TestWaitForWithClosedChannel tests WaitFor when it receives a closed channel. The WaitFor should // always return ErrWaitTimeout. </ins> func TestWaitForWithClosedChannel(t *testing.T) { stopCh := make(chan struct{}) close(stopCh) <ins> c := make(chan struct{}) defer close(c) </ins> start := time.Now() <del> err := WaitFor(poller(ForeverTestTimeout, ForeverTestTimeout), func() (bool, error) { </del> <ins> err := WaitFor(func(done <-chan struct{}) <-chan struct{} { return c }, func() (bool, error) { </ins> return false, nil }, stopCh) duration := time.Now().Sub(start)

Users of the function would expect the close of the "done" channel should terminate the . We should change the function to align to users' expectation. We should document what the channel does. This is a behavior change, to be cautious, we should add a release note claiming breaking change. /kind bug
/sig api-machinery cc Feel free to assign to yourself if you have time. Otherwise I'll attempt a fix tomorrow ;)
I'll fix it. /assign
GitHub didn't allow me to assign the following users: kdada. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : I'll fix it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: return &etcd.Response{}, EtcdErrorNotFound } f.t.Logf("returning %v: %#v %#v", key, result.R, result.E) <ins> // Sort response, note this will alter resutl.R. if result.R.Node != nil && result.R.Node.Nodes != nil && sort { f.sortResponse(result.R.Node.Nodes) } </ins> return result.R, result.E } <ins> func (f *FakeEtcdClient) sortResponse(nodes etcd.Nodes) { for i := range nodes { if nodes[i].Dir { f.sortResponse(nodes[i].Nodes) } } sort.Sort(nodes) } </ins> func (f *FakeEtcdClient) nodeExists(key string) bool { result, ok := f.Data[key] return ok && result.R != nil && result.R.Node != nil && result.E == nil

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" klabels "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait"

[X] Test PR: kuberenetes/kubernetes 17 June 2021 [x] Two weeks soak end date: 1 Jul 2021 [X] Test promotion PR: kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatefulSet endpoints which are untested. Create a watch to track stateful set events Create a stateful set with a static label. Confirm that the pods are running. Get the stateful set status. Parse the response and confirm that the stateful set status conditions can be listed. Update the stateful set status. Confirm via the watch that the status has been updated. Patch the stateful set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for stateful set lifecycle test. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return ctx.Err() } <del> if cmderr != nil { if mounter.withSafeNotMountedBehavior && strings.Contains(string(out), errNotMounted) { klog.V(4).Infof("ignoring 'not mounted' error for %s", path) return nil } return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", cmderr, path, string(out)) </del> <ins> if err != nil { return checkUmountError(target, command, output, err, withSafeNotMountedBehavior) </ins> } return nil } <del> func forceUmount(path string) error { cmd := exec.Command("umount", "-f", path) out, cmderr := cmd.CombinedOutput() </del> <ins> func forceUmount(target string, withSafeNotMountedBehavior bool) error { command := exec.Command("umount", "-f", target) output, err := command.CombinedOutput() </ins> <del> if cmderr != nil { return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", cmderr, path, string(out)) </del> <ins> if err != nil { return checkUmountError(target, command, output, err, withSafeNotMountedBehavior) </ins> } return nil } <ins> // checkUmountError checks a result of umount command and determine a return value. func checkUmountError(target string, command *exec.Cmd, output []byte, err error, withSafeNotMountedBehavior bool) error { if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } if withSafeNotMountedBehavior && strings.Contains(string(output), errNotMounted) { klog.V(4).Infof("ignoring 'not mounted' error for %s", target) return nil } return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", err, target, string(output)) } </ins>

A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


Question: "resources but no selectors": { args: []string{"pods"}, errFn: func(err error) bool { <del> return strings.Contains(err.Error(), "resource(s) were provided, but no name, label selector, or --all flag specified") </del> <ins> return strings.Contains(err.Error(), "resource(s) were provided, but no name was specified") </ins> }, }, "multiple resources but no selectors": { args: []string{"pods,deployments"}, errFn: func(err error) bool { <del> return strings.Contains(err.Error(), "resource(s) were provided, but no name, label selector, or --all flag specified") </del> <ins> return strings.Contains(err.Error(), "resource(s) were provided, but no name was specified") </ins> }, }, }

What happened: Kubectl patch sub-command shows wrong error help message if an input is wrong. The error message says and when I try to use flag it says . What you expected to happen: I expected either the first error message to not have or should have a flag . How to reproduce it (as minimally and precisely as possible): If you run the following command without a node name you get this error: As the above error said you can specify , you specify that. This fails with another error saying is unknown flag: And in the help there is no flag: Environment: Kubernetes version (use ):
/sig cli
I think we need to decide whether to add flag --all and --selector to kubectl patch. After the decision, i can help.
/assign
on my queue, will look tomorrow :D
/triage accepted Probably just need to update error message here.
oh sorry, missed that one. Will look now
/assign
OK, so we decided to correct the error message per the PR right? :)
as you're dealing with this PR, I'll left you as assignee and follow the PR review :) /unassign


Question: // createTags calls EC2 CreateTags, but adds retry-on-failure logic // We retry mainly because if we create an object, we cannot tag it until it is "fully created" (eventual consistency) // The error code varies though (depending on what we are tagging), so we simply retry on all errors <del> func (s *AWSCloud) createTags(request *ec2.CreateTagsInput) (*ec2.CreateTagsOutput, error) { </del> <ins> func (s *AWSCloud) createTags(resourceID string, tags map[string]string) error { if tags == nil || len(tags) == 0 { return nil } var awsTags []*ec2.Tag for k, v := range tags { tag := &ec2.Tag{ Key:   aws.String(k), Value: aws.String(v), } awsTags = append(awsTags, tag) } request := &ec2.CreateTagsInput{} request.Resources = []*string{&resourceID} request.Tags = awsTags </ins> // TODO: We really should do exponential backoff here attempt := 0 maxAttempts := 60 for { <del> response, err := s.ec2.CreateTags(request) </del> <ins> _, err := s.ec2.CreateTags(request) </ins> if err == nil { <del> return response, err </del> <ins> return nil </ins> } // We could check that the error is retryable, but the error code changes based on what we are tagging

After setting up a few services, I noticed the following output in : I assume the service tried to configure an ELB in the past (not sure when or how or what details it tried to use) and has kept trying and now can't even use the AWS API. How can I either: 1) tell it to please stop so that I can use my AWS account again 2) fix it so that it does configure the ELB /cc
Sorry about the problem. Looks like we need a sleep / backoff policy here! To get you up and running again, if you redeploy the nginx service without Type=LoadBalancer, we will (should) stop trying to create a load balancer, and you should get out of the AWS API penalty-box. Sorry about this, and thanks for the report. Looks like this is pretty important to fix. Do you happen to know why you can't create an ELB? My guess is that you have hit your quota?
I'll remove the service, wait for the penalty-box to expire, then build it again so that we can see what the real error was.
Adding the LoadBalancer service back in, I'm seeing:
I took a few different steps and seem to have gotten it both working in different ways: First I deleted the service because it was erroring, but I noticed the count was still going up. So I deleted the security group it said was a duplicate and then it finished setting the ELB up! That meant I needed to manually clean up the ELB resources and security groups because k8 thought it was already deleted. After cleaning up all the resources in AWS again, I the service again and it seemed to work fine! I'm not sure how/why it initially confused itself with the duplicate SG, but it doesn't necessarily seem to happen every time. The code could likely just, in the event that it sees the SG already exists, continue on with the setup. I could send you a PR to handle that if you'd like!
I have a PR that I think covers this: . I thought it was a one-off error, but it sounds like it wasn't. Thanks so much for pausing & retrying also - that was super-helpful. Even if we think covers this, let's leave this issue open to track the need for a backoff to avoid overloading the AWS API.

(And presumably this aggressive retrying happens on other clouds also, though maybe they have overload a sleep built in).
Certainly looks like should fix it. I'm still wondering why I'd see it failing literally for hours in a loop though. If it was a problem with AWS being eventually consistent, then after an hour I'd think that first check, looking to see if the securityGroup existed, would start to return the group! I don't know the k8 code well yet, though so perhaps it was looping in a different place that wasn't running that code to see if the security group already existed.
you're right; my explanation wasn't entirely correct... So security group names are unique per VPC. We also have a per-cluster name, and we filter all our AWS API interactions to only select resources tagged with this name. If we had a security group name collision across two k8s clusters in the same VPC, that would be bad. Thankfully the security group name we're creating includes a UUID, so this "won't happen". However, the problem is that we created the security group, and then failed to tag it. But we were filtering based on our cluster. So our "describe security groups" operation would never see the SG, because it wasn't appropriately tagged. We have a few workarounds to make this much less likely, and I have opened the "big picture" issue . And I'm going to think about whether we should in fact be filtering when we DescribeSecurityGroups - I think the answer is no in the case where the name has a UUID in it.
Ah! That totally makes sense. The race between resource creation and resource tagging can created "invisible" resources if the process of tagging fails. Maybe an anti-entropy function that can look at all the resources and normalize them. In the case of invisible resources, you can just delete them and hope that recreating them won't fail the tagging process.
I see this as well when frequently deploying/testing. Unfortunately, it happens far too often (running v1.0.0).
+1
I've been having a similar issue to this, Kubernetes can't create a load balancer as it says the security group already exists. The workaround I've found is to manually delete the security group and wait for Kubernetes to successfully create it again.

I've generally had the problem when quickly deleting a service with a and then creating it again (I see now I should use ). I wonder if it is something to do with AWS reassigning the same load balancer back to you if you haven't finished deleting it coupled with the tagging issue mentioned earlier.
+1
cc
I had an email discussion with on this and he said that what's remaining shouldn't be considered a blocker for 1.1, so I'll move it to 1.2-candidate.
Can you summarize what's remaining to do for this issue? Should we just close it in favor of ?
I see you the v1.2 milestone -- will you be able to look into this in the next few days?


Question: t.Run(tc.name, func(t *testing.T) { // Set up a backend server backendServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { <ins> if tc.redirect != "" { w.Header().Set("Location", tc.redirect) } </ins> w.WriteHeader(tc.serverStatusCode) w.Write(originalBody) }))

Upgraded from Kubernetes v1.25.0 to v1.25.1. Standard metrics-server deployment now causes the following message to appear in the kube-apiserver logs on a recurring basis: Adding a quick hack debug Infof print shows: It appears that the apiserver is sending an If-None-Match cache control header when refreshing the openapi spec, and the metrics-server service is correctly responding with a 304 Not Modified. No errors logged; proper refresh of OpenAPI specs Deploy metrics-server, or any other service that uses apiserver aggregation and honors caching headers such as , to a Kubernetes 1.25.1 cluster. made bad assumptions; not all HTTP 3XX responses are redirects. It should probably only reject the response if it is 300-399 AND the header is set. <details</detailsnone <details</detailsk3s containerd flannel
Taking a stab at some tags here... /cc /cc /cc /area apiserver /sig api-machinery
thanks for the report, looks like this should be /priority critical-urgent /kind regression cc since this requires a respin of a final 1.22 patch release :-/
Fix open in , will get cherry-picks open to start greening up CI
/triage accepted
I need to make a redirect to the external URL but it's not allowed :(


Question: "STANDARD_D3_V2":         16, "STANDARD_D3_V2_PROMO":   16, "STANDARD_D4":            32, <ins> "STANDARD_D48AS_V4":      32, "STANDARD_D48A_V4":       32, </ins> "STANDARD_D48S_V3":       32, "STANDARD_D48_V3":        32, "STANDARD_D4AS_V4":       8,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: controllerManager.Run(10 * time.Minute) // Kubelet (localhost) <ins> os.MkdirAll(testRootDir, 0750) </ins> cfg1 := config.NewPodConfig(config.PodConfigNotificationSnapshotAndUpdates) config.NewSourceEtcd(config.EtcdKeyForHost(machineList[0]), etcdClient, cfg1.Channel("etcd")) config.NewSourceURL(manifestURL, 5*time.Second, cfg1.Channel("url")) <del> myKubelet := kubelet.NewIntegrationTestKubelet(machineList[0], &fakeDocker1) </del> <ins> myKubelet := kubelet.NewIntegrationTestKubelet(machineList[0], testRootDir, &fakeDocker1) </ins> go util.Forever(func() { myKubelet.Run(cfg1.Updates()) }, 0) go util.Forever(func() { kubelet.ListenAndServeKubeletServer(myKubelet, cfg1.Channel("http"), "localhost", 10250)

E1003 10:53:11. ] Could not read directory: , (open : no such file or directory) This happens many times


Question: // Exec runs the kubectl executable. func (b KubectlBuilder) Exec() (string, error) { <ins> stdout, _, err := b.ExecWithFullOutput() return stdout, err } // ExecWithFullOutput runs the kubectl executable, and returns the stdout and stderr. func (b KubectlBuilder) ExecWithFullOutput() (string, string, error) { </ins> var stdout, stderr bytes.Buffer cmd := b.cmd cmd.Stdout, cmd.Stderr = &stdout, &stderr Logf("Running '%s %s'", cmd.Path, strings.Join(cmd.Args[1:], " ")) // skip arg[0] as it is printed separately if err := cmd.Start(); err != nil { <del> return "", fmt.Errorf("error starting %v:nCommand stdout:n%vnstderr:n%vnerror:n%v", cmd, cmd.Stdout, cmd.Stderr, err) </del> <ins> return "", "", fmt.Errorf("error starting %v:nCommand stdout:n%vnstderr:n%vnerror:n%v", cmd, cmd.Stdout, cmd.Stderr, err) </ins> } errCh := make(chan error, 1) go func() {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ]

externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ]

externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows


Question: return utilerrors.NewAggregate(errs) } <del> // Gets the (CPU) container the specified pid is in. </del> <ins> // getContainer returns the cgroup associated with the specified pid. // It enforces a unified hierarchy for memory and cpu cgroups. // On systemd environments, it uses the name=systemd cgroup for the specified pid. </ins> func getContainer(pid int) (string, error) { cgs, err := cgroups.ParseCgroupFile(fmt.Sprintf("/proc/%d/cgroup", pid)) if err != nil { return "", err } <del> cg, ok := cgs["cpu"] if ok { return cg, nil </del> <ins> cpu, found := cgs["cpu"] if !found { return "", cgroups.NewNotFoundError("cpu") } memory, found := cgs["memory"] if !found { return "", cgroups.NewNotFoundError("memory") } // since we use this container for accounting, we need to ensure its a unified hierarchy. if cpu != memory { return "", fmt.Errorf("cpu and memory cgroup hierarchy not unified.  cpu: %s, memory: %s", cpu, memory) } // on systemd, every pid is in a unified cgroup hierarchy (name=systemd as seen in systemd-cgls) // cpu and memory accounting is off by default, users may choose to enable it per unit or globally. // users could enable CPU and memory accounting globally via /etc/systemd/system.conf (DefaultCPUAccounting=true DefaultMemoryAccounting=true). // users could also enable CPU and memory accounting per unit via CPUAccounting=true and MemoryAccounting=true // we only warn if accounting is not enabled for CPU or memory so as to not break local development flows where kubelet is launched in a terminal. // for example, the cgroup for the user session will be something like /user.slice/user-X.slice/session-X.scope, but the cpu and memory // cgroup will be the closest ancestor where accounting is performed (most likely /) on systems that launch docker containers. // as a result, on those systems, you will not get cpu or memory accounting statistics for kubelet. // in addition, you would not get memory or cpu accounting for the runtime unless accounting was enabled on its unit (or globally). if systemd, found := cgs["name=systemd"]; found { if systemd != cpu { glog.Warningf("CPUAccounting not enabled for pid: %d", pid) } if systemd != memory { glog.Warningf("MemoryAccounting not enabled for pid: %d", pid) } return systemd, nil </ins> } <del> return "", cgroups.NewNotFoundError("cpu") </del> <ins> return cpu, nil </ins> } // Ensures the system container is created and all non-kernel threads and process 1

container manager setup node routine should ensure that cpu and memory cgroups for container runtime are of the same value, and warn if they are equal to . For background, on systemd environments, the default unit file for has CPU and Memory accounting not enabled, so the /proc/.../cgroup file reports as cpu cgroup.
/cc - I will send a bug-fix for 1.3 for this in next couple days.
/cc
Looking forward to the fix
The PR that fixes this issue was prematurely merged by the merge bot so re-opening.


Question: } e2elog.Logf("Got an error creating template %d: %v", i, err) } <del> ginkgo.Fail("Unable to create template %d, exiting", i) </del> <ins> e2elog.Fail("Unable to create template %d, exiting", i) </ins> }) })

/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign


Question: "k8s.io/component-base/metrics/legacyregistry" "k8s.io/component-base/metrics/testutil" testingclock "k8s.io/utils/clock/testing" <ins> clientmetrics "sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/metrics" ccmetrics "sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/common/metrics" "sigs.k8s.io/apiserver-network-proxy/konnectivity-client/proto/client" </ins> ) type fakeEgressSelection struct {

At (originally ) component-base metrics exposed Registerer API for raw prometheus registration, and I took dependency in The resulting does not seem to emit the new metrics (example: ). The metrics handler should emit the registered metrics. Falling back to does work. So there is some material difference with . 24, 1.25, 1.26 GKE No response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign jkh52


Question: } // TransformFunc allows for transforming an object before it will be processed. <del> // TransformFunc (similarly to ResourceEventHandler functions) should be able // to correctly handle the tombstone of type cache.DeletedFinalStateUnknown. // // New in v1.27: In such cases, the contained object will already have gone // through the transform object separately (when it was added / updated prior // to the delete), so the TransformFunc can likely safely ignore such objects // (i.e., just return the input object). </del> // // The most common usage pattern is to clean-up some parts of the object to // reduce component memory usage if a given component doesn't care about them. // <del> // New in v1.27: unless the object is a DeletedFinalStateUnknown, TransformFunc // sees the object before any other actor, and it is now safe to mutate the // object in place instead of making a copy. </del> <ins> // New in v1.27: TransformFunc sees the object before any other actor, and it // is now safe to mutate the object in place instead of making a copy. // // It's recommended for the TransformFunc to be idempotent. // It MUST be idempotent if objects already present in the cache are passed to // the Replace() to avoid re-mutating them. Default informers do not pass // existing objects to Replace though. </ins> // // Note that TransformFunc is called while inserting objects into the // notification queue and is therefore extremely performance sensitive; please

We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used.

FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted


Question: "fmt" "io" "net/http" <ins> "reflect" </ins> "testing" "github.com/spf13/cobra"

When running , if you don't specify the deprecated and instead just do , it doesn't strip off the 1st command line arg, so sends the command to the container, whereas correctly sends just Regression from
FYI
We still support --, correct?
yes, we still support , but it's marked deprecated.
cc
has the fix. (and better unit test coverage) Sorry for the breakage!


Question: for _, infoClaim := range cache.claimInfo { claimInfoStateList = append(claimInfoStateList, infoClaim.ClaimInfoState) } <del> return cache.state.Store(claimInfoStateList) </del> <ins> checkpoint, err := state.NewCheckpoint(claimInfoStateList) if err != nil { return err } return cache.checkpointer.Store(checkpoint) </ins> } // cdiDevicesAsList returns a list of CDIDevices from the provided claim info.

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: PodTemplate: validPodTemplate, }, }, <ins> "selector_doesnt_match": { JSONBase: JSONBase{ID: "abc"}, DesiredState: ReplicationControllerState{ ReplicaSelector: map[string]string{"foo": "bar"}, PodTemplate:     validPodTemplate, }, }, </ins> "invalid manifest": { JSONBase: JSONBase{ID: "abc"}, DesiredState: ReplicationControllerState{

If a ReplicaController has a selector that doesn't match the template it'll just create tons and tons of pods. It should validate that the pods it is creating have a good chance of satisfying its selector.
It would also be useful to validate that the selector doesn't obviously overlap with selectors of other replicationControllers.


Question: ds, err = c.AppsV1().DaemonSets(ns).Patch(dsName, types.StrategicMergePatchType, []byte(patch)) Expect(err).NotTo(HaveOccurred()) <ins> // Time to complete the rolling upgrade is proportional to the number of nodes in the cluster. // Get the number of nodes, and set the timeout appropriately. nodes, err := c.CoreV1().Nodes().List(metav1.ListOptions{}) Expect(err).NotTo(HaveOccurred()) nodeCount := len(nodes.Items) retryTimeout := dsRetryTimeout + time.Duration(nodeCount*30)*time.Second </ins> By("Check that daemon pods images are updated.") <del> err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkDaemonPodsImageAndAvailability(c, ds, RedisImage, 1)) </del> <ins> err = wait.PollImmediate(dsRetryPeriod, retryTimeout, checkDaemonPodsImageAndAvailability(c, ds, RedisImage, 1)) </ins> Expect(err).NotTo(HaveOccurred()) By("Check that daemon pods are still running on every node of the cluster.")

When running conformance testing on a cluster with 8 nodes schedulable for jobs, the conformance test for rolling updates on daemon sets fails due to timeout. Test name: Test code is here: It utilizes a hardcoded timeout limit of 5m. What happened: The logs seem to indicate nothing wrong other than the time it takes to complete the rolling restart. The rolling restart timeout is hardcoded at 5m which doesn't seem to be enough time for large enough clusters. What you expected to happen: I would expect the timeout to either be sliding or for the check to be kicking the can; ensuring that the update is proceeding by at least one node every minute or two. How to reproduce it (as minimally and precisely as possible): Requires a cluster with at least 8 schedulable nodes; more would make the timeout more certain. Anything else we need to know?: I was trying to see what k8s was testing itself with and it seems (according to this log) that the test is run against a cluster with 4 schedulable nodes and takes ~3m which would line up with what I'm seeing (8 nodes for ~6m, 1 more than given by the hardcoded deadline) Seems like there are a few options: use a "kick the can" strategy to ensure the timeout scales use a timeout calculated from the number of schedulable nodes (mitigation) increase the max number of down nodes so the update can occur faster Environment: Kubernetes version (use ): From test logs (cluster no longer available): /sig testing /kind bug
If there is agreement on the issue, I'm happy to implement the fix for this. I couldn't find anyone else really complaining about this timeout though so I wanted to be sure to get a sanity check first though.
Can I get someone to confirm/debunk my analysis above?
I am running into this as well. It seems like the first image () that is being used in the test takes around 30-40 seconds to be terminated. The test consistently hits the 5 minute timeout on a 10 node cluster, given that rolling all the pods takes longer than 300 seconds. The binary [sleeps for 60 seconds][0] after getting the SIGTERM signal, which I believe would cause the kubelet to force-kill the process after the default 30 second grace period. Not sure why that sleep is there.

[0]:
i can repro this in kubeadm-dind with It's a simple timeout issue. The behavior has changed between 1.11 and 1.12 in 1.11 this test takes less than 5 minutes for 10 nodes in 1.12 the test is now failing consistently
I don't know if I'm going to have enough bandwidth on this right now to dig into this. It's likely the shift from controller<scheduler on daemonsets. I don't know if/when that landed. It seems kind of strange that the test takes that long TBH.
I think the issue is in the test image that we are using, and not necessarily in the controller/scheduler. Seems like there was a sleep introduced to the serve-hostname process as part of this PR: I am wondering if it was mistakenly left in there. If the sleep weren't there, the workload should exit immediately, allowing the rolling upgrade of the daemonset pods to complete within the allocated timeout.
/area conformance
I am working on a PR that removes the sleep from . In the same PR, I am planning to bump the image version, and update all references I can find to the image.
Thanks for all the eyes looking at this, I appreciate it. Removing the sleep will obviously speed things up and probably pass in these use cases. It still seems like something should be done so that the timeout scales though; if we have 1k node clusters we'd probably still have a problem, yes?
/assign We're going to need an image push update for conformance regression. WRT timeout scaling... yes, but let's base it on some data and experiments.
The sleep in there was over a year ago. What changed RECENTLY? Curiously there is a serve-hostname v1.4 tagged on already, but I can;t say where it came from, since VERSION in k/k/test/images/serve-hostname says 1.2 ... That sleep was to test graceful termination, I think. removing it may have side-effects. Can we make the test runs set ? Or maybe we make this termination-sleep a flag? If we change it at all, we should bump the version to 1.5 and supercede all others.
Thanks The most recent, related change I could happened 6 months ago: (PR ).

With this change, the conformance test started using the 1.1 version of the image, which includes the sleep. I think it'd be great to know if and how this sleep is being used in tests. also suggested (in ) using a flag to control the sleep behavior, so perhaps has more details about how the sleep is used. With that said, it seems like we could address the immediate issue by setting the termination grace period to 1 or 2 seconds in the DaemonSet rolling upgrade test. et al. - Would love your thoughts on whether this is an acceptable fix.
cc
cc
Agree this needs fixing Will defer to DaemonSet owners on correct route to fix, but I like suggestion. Perhaps we can just use a smaller sleep.. I see in the linked PR that we're bumping to 1.3 despite the mystery 1.4 image, we should bump to 1.5 per other point
What images we use for this test shouldn't matter, as we only care about whether the images is updated. So if serve-hostname is causing us problem we can just use some other images instead. We could also increase the timeout based on the number of nodes for DaemonSet rolling update test.
We have 45 nodes and we are unable to get this test to pass as well due to timeouts. Changing the image might help, but maybe adjusting the timeout based on node count is a solution as well.
totally. suggested that as well and we have it to the PR