Question: "v":                   true, "vmodule":             true, "log-flush-frequency": true, <ins> "provider-id":         true, </ins> } fs.VisitAll(func(f *pflag.Flag) { if notDeprecated[f.Name] {

Un-deperecate the kubelet flag, as this option is typically set per-node similar to (which is not deprecated). Or more generally, consider not deprecating fields available in config that make sense to set per-node. I noticed this while trying to ensure KIND moves towards not setting any deprecated flags, this one is tricky because it's not a fixed value. For a lot of kubernetes users, kubeadm is used for bootstrapping, and kubeadm supports supplying additional kubelet flags when configuring each node, but the kubelet config is cluster-wide, created during setting up the first node and uploaded to a configmap then used for all nodes. In the future kubeadm may have the ability to do per-node kubelet config, but that effort will probably be involved and isn't staffed at the moment. technically you could run kubeadm in phases and then modify the config, but this is more complex to implement. It's possible this won't be necessary if kubeadm patches can be extended to kubelet which may be a smaller feature change for kubeadm, but personally I think being able to configure simple per-node strings with a flag override is desirable for implementing clusters, and we've had the flag for many years with no trouble ... The actual deprecation seems to be due to a blanket deprecation on things possible to set in config, and not targeted at this flag in particular. See discussion in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
technically KIND doesn't strictly need this flag, FWIW, but we were asked by users to set it to make detection easier and have values more similar to "real" clusters. other projects in Kubernetes are also doing this, e.g. some of of the cluster api-provider implementations:
I think for 1.25 in kubeadm we can enable patching of kubeletconfuguration.

This will allow us to use the current global kubletconfiguration for all nodes, but patches can act as the node instance specific config. Flags would still take precedence over patches and are the simpler UX. But if one day all flags are removed the above solution will still support the instance specific use case. I don't know whether some flags should be de-deprecated, but IIRC the plan to migrate all flags to kubeletconfiguration and not add more flags was still desirable by sig node.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign It seems to be reasonable. I opened to undeprecate it.


Question: } lun := strconv.Itoa(iscsi.Lun) <ins> portal := portalBuilder(iscsi.TargetPortal) </ins> return &iscsiDiskBuilder{ iscsiDisk: &iscsiDisk{ podUID:  podUID, volName: spec.Name(), <del> portal:  iscsi.TargetPortal, </del> <ins> portal:  portal, </ins> iqn:     iscsi.IQN, lun:     lun, manager: manager, <del> mounter: mounter, </del> <ins> mounter: &mount.SafeFormatAndMount{mounter, exec.New()}, </ins> plugin:  plugin}, fsType:   iscsi.FSType, readOnly: readOnly,

Documentation says that port number for is optional, but I've found that not to be true. It's due to the way it detects whether the disk is attached or not. also must be an IP. Refer to:
/cc
I see what you meant. Good catch. According to , the disk path contains both ip and port. So port is indeed required even for default port 3260. If you don't have a pr yet, I'll provide a fix to the doc.
I don't have a CLA in file yet, so I'll leave it up to you. Thanks.
check this out


Question: default: // We lost the lock. klog.ErrorS(nil, "Leaderelection lost") <del> os.Exit(1) </del> <ins> klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> } }, }

kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.


Question: func createWindowsJobObject(pc uint32) (windows.Handle, error) { job, err := windows.CreateJobObject(nil, nil) if err != nil { <del> return windows.InvalidHandle, errors.Wrap(err, "windows.CreateJobObject failed") </del> <ins> return windows.InvalidHandle, fmt.Errorf("windows.CreateJobObject failed: %w", err) </ins> } limitInfo := windows.JOBOBJECT_BASIC_LIMIT_INFORMATION{ LimitFlags:    windows.JOB_OBJECT_LIMIT_PRIORITY_CLASS,

Why is this needed? The functionality that the package provides is already present in golang's builtin errors (using in ) and the github repo is now read-only. /kind cleanup /priority backlog /good-first-issue Notes to people that want to collaborate Avoid creating a single PR for the entire repo. Create multiple PRs, each targeting a specific SIG. Do not submit PRs for
/area code-organization
/kind cleanup
this is only partially true, e.g. %w does not provide stack traces
/assign
I see. Should we close this issue then?
I think it depends on how it's being used. If the traces are not the goal then we can do a simple substitution. It's probably best to drop read only dependencies ASAP but it's worth noting that if we depended on the full functionality in any way we'll need a more complex replacement (KIND did/does, not sure about k/k)
k/k git log suggests that some cleanup has already been done in the past (maybe in other commits too): This issue triggered 3 PRs and 2 of them are merged. there does not seem to be calls to so most cases it looks it's just wrapping errors.
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
I see. Added a note.
is it still open ?
should be replaced with the standard library for the following reasons: archived library with no maintenance or security updates the go community used pkg/errors to introduce eventually into the standard library hence the approach that was selected is more idiomatic stack traces are hardly used in error wrapping and are not a reason not to refactor
On place left in Except the k/k repo, as is archived, some other repos may need an update.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is it necessary to remove "" from every location? I noticed it in many places. Also, do I need to create separate PRs for each file where I remove it?
/assign
From kuberentes/kubernetes repo view, only two locations of kubectl code should be updated. (Do not submit PRs for ) see For other repos, we should follow up until the dependencies' repo is updated. I will track them in
/assign
/assign
to remove dependency you can use the standard Go error package or creating your own handling package, this could contain custom functions for generating and handling errors.
let me work on this issue now please /assign
We are in code freeze. I suggest waiting for a couple of weeks.
Hello, is this still up?
Let's get this going
/assign
Just opened a PR for this issue
/assign
Update the remaining todo items.
/sig architecture umbrella / code org
/assign
Hey Are you still working on this one? If not- would you mind if I take a stab at it? Thanks!
Is this issue really still open? When I do I only get hits for cmd/kubeadm. Am I missing something?
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { framework.Failf("Failed to list all dns service pods") } <del> if len(dnsPods.Items) != 1 { framework.Failf("Unexpected number of pods (%d) matches the label selector %v", len(dnsPods.Items), dnsServiceLabelSelector.String()) </del> <ins> if len(dnsPods.Items) < 1 { framework.Failf("No pods match the label selector %v", dnsServiceLabelSelector.String()) </ins> } framework.ExpectNoError(framework.WaitForPodRunningInNamespace(f.Client, dnsPods.Items[0].Name, api.NamespaceSystem)) }

everything with is now failing, likely after 2021-11-04 17:44 PDT No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The best course of actions is likely just remove those tests from the job, similar to


Question: "//pkg/registry/generic/registry:go_default_library", "//pkg/runtime:go_default_library", "//pkg/storage:go_default_library", <ins> "//pkg/util/wait:go_default_library", </ins> ], )

What happened: When I try to test after implementing the 1st approach according to , I found pod disruption budgets cannot handle concurrent requests. The error message is: It is an Optimistic Lock error message. It means PDB can't handle concurrent requests. How to reproduce it (as minimally and precisely as possible): 1) Create a deployment with 6 replicas and schedule them to the same node, say node A. You can leave only one node scheduling enabled, and then create the deployment. 2) Create the PDB that match the deployment's label 3) Uncordon another node, say node B 4) Drain node A using concurrent POST eviction requests. Ref: I'm not sure if this behavior is OK for PDB. I think I can make work by waiting for a while if current pod label(s) match previous pod label(s).
Oh, that's interesting. I think the problem is that this function doesn't retry if UpdateStatus() fails due to version conflict. I think the solution is to modify the caller, i.e. this function so that instead of just it instead does something like This will loop re-doing checkAndDecrement() and getPodDisruptionBudgets() until checkAndDecrement() succeeds, or fails with a non-conflict error. Does that make sense? cc/ for sanity check
Or use PATCH?
Yes, PATCH should work also. I haven't written code that uses PATCH, so I can't advise exactly the code to write here (there are examples by grepping for patch, but I'm not sure exactly what is the right way to do it from inside of API server code, which is where we are here). can you do a PR to implement this, either using the approach I described above or using PATCH? I think this is important to get into 1.5.
I think PATCH may NOT be safe here. For example: 1) current pdb 2) creating eviction A: get 3) creating eviction B: get 4) creating eviction A: PATCH pdb 5) creating eviction B: PATCH pdb 6) pdb's . It's wrong. Retry loop make more sense to me.
Good point! You're right. Retry loop should be OK.
I can help to create a PR using retry loop.

Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
Done.
who is actually working on this issue? Could one of you set yourselves as an assignee for this issue? Thanks! cc:
retry logic with PR needs reviews/lgtm


Question: // utils.go contains functions used accross test suites. const ( <del> cniRelease   = "07a8a28637e97b22eb8dfe710eeae1344f69d16e" </del> <ins> cniRelease   = "0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff" </ins> cniDirectory = "cni" cniURL       = "https://storage.googleapis.com/kubernetes-release/network-plugins/cni-" + cniRelease + ".tar.gz" )

Started as a tangent of Many CNI plugins don't properly handle a action on missing resources. The spec says, , however many improperly return an error. I see the steps moving forward to be: [x] Modify CNI plugins bundled with Kubernetes to properly support del (flannel already done; bridge, ipvlan, macvlan, and ptp in containernetworking/cni) [x] Create release of containernetworking/cni Does this make sense? cc/
SGTM - I'm working on the right now and can do the CNI release too. On the 3rd check box - why is that still a thing? Doesn't the CNI project provide the required artifacts as part of its release? Maybe not something to resolve now but I'd like to make sure the CNI project is providing the right things to projects that use it.
I just want to say thanks to everyone for jumping on this.
status tracking doc
I can test out patches as needed. I currently only have test cases for Weave and Calico, but it should not be too difficult to add others.
It would be great to get flannel tested too. You probably need a master build of the flannel CNI plugin and possibly some flannel patches to be merged. Feel free to DM on slack if you need more info.
s390x builds are missing still. If that's , we can switch to v0.5.1 in time for v1.7. However, I really don't want to bump from v0.3.0-ish to v0.5.0 one day before release. This is a v1.7 issue to ensure all CNI plugins (including the official) work as intended with Kubernetes and an issue to make sure we update to CNI v0.5.0 w/ spec v0.3.0 in time for v1.7. I think will solve the short-term solutions. Also it seems like Weave, Calico and maybe Canal has fixed this as-is. I think Flannel works with
I'm a little confused! You mean s390x builds for flanneld? And what do you mean by This is a v1.7 issue I thought this was a 1.6 issue?

CNI still needs s390x binaries to be consumable by k8s: Two other conclusions: 1) We should go with as the immediate solution 2) We should upgrade to CNI v0.5.1 (when released( officially for Kubernetes in time for v1.7 and enforce the latest spec (v0.3.0) for all network plugins
We should get the PR in (after testing), but this is not a really solution as it only avoids hitting this problem at cluster setup.
OK, that makes sense on the s390x binaries. I've just released v0.5.1 of CNI, which updates the plugins with the new del behaviour. They should all still be back compatible with the older spec versions so I recommend that K8s takes this CNI release.
I'm working on a patch for that bump now
Be aware that has just been merged. If you sync to HEAD, you won't be able to reproduce this problem at cluster setup easily.
New cni packaged tarball (based on v0.5.1) has been pushed:
-- can this be closed?
we can't close until all the consumers have been updated to point to the new tarballs
What is the ETA for resolving this?
Ok, I see that is at the top of the submit queue.


Question: return err } default: <del> return fmt.Errorf("pod %v state wasn't added but get updated", key) </del> <ins> return fmt.Errorf("pod %v is not added to scheduler cache, so cannot be updated", key) </ins> } return nil }

/kind bug There are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via Events. I noticed these two in particular but there may be others: Whether or not these should actually be surfacing as events, the messages should be cleaned up.
cc
Seems that these are internal or implement details in scheduler, do our end users can fully understand what the events really mean? I think it is depend on our own implement. idea?
I believe, the problem is with in the message but I am not able to reproduce it, can you please share steps to reproduce this.
No unfortunately, one of our users reported seeing it when there system was in a bad state, not in a reproducible way. The point of the issue is just that the grammar in those messages is bad, which should be fixable even if you can't get them to fire.


Question: // resourceVersion may be used to specify what version to begin watching, // which should be the current resourceVersion, and no longer rv+1 // (e.g. reconnecting without missing any updates). <ins> // If resource version is "0", this interface will list current objects directory defined by key // and send them in "ADDED" events, before watch starts. </ins> WatchList(ctx context.Context, key string, resourceVersion string, p SelectionPredicate) (watch.Interface, error) // Get unmarshals json found at key into objPtr. On a not found error, will either

Currently, Watch request with resourceVersion=0 results in: returning a single "watch event" for every object existing "now" starts watching from that point This is pretty misleading (e.g. it's different that what we have in etcd, where there is no first step from above). In the v2 api, we should probably eliminate the first stop (or even disallow watch requests with resourceVersion = 0).
/cc
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/remove-lifecycle rotten /lifecycle frozen
What do you mean by v2 API? Are we ever going to have a v2 of the machiery and semantics?
In 2015 I thought it's not ridiculous. I'm not sure I believe in it now. Moreover, with we will actually be taking benefits of semantic like this, so we should close this.


Question: podIndexLabelDisabled:  true, expectedReady:          ptr.To[int32](0), }, <ins> "FailureTarget=False condition added manually is ignored": { jobPodFailurePolicy: true, parallelism:         1, completions:         1, activePods:          1, readyPods:           1, initialStatus: &jobInitialStatus{ active: 1, startTime: func() *time.Time { now := time.Now() return &now }(), conditions: []batch.JobCondition{ { Type:    batch.JobFailureTarget, Status:  v1.ConditionFalse, Reason:  "ConditionAddedManually", Message: "Testing", }, }, }, expectedActive: 1, expectedReady:  ptr.To[int32](1), expectedConditions: []batch.JobCondition{ { Type:    batch.JobFailureTarget, Status:  v1.ConditionFalse, Reason:  "ConditionAddedManually", Message: "Testing", }, }, }, </ins> } for name, tc := range testCases {

When the FailureTarget=False condition is manually, then the Job terminates. The condition should be ignored by the Job controller. the like this: the following script to repro: Issue: the Job gets terminated with the status like this: No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign Thanks!
/sig apps


Question: v1 "k8s.io/api/core/v1" storagev1 "k8s.io/api/storage/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/apimachinery/pkg/watch" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/informers"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are some places in that wait for conditions . We should update all the places to set something like /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: There are some places in that wait for conditions . We should update all the places to set something like /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /remove-kind bug
I can try to fix this :) /assign
/sig storage


Question: --secure-port=443  --basic-auth-file=/srv/kubernetes/basic_auth.csv  --target-ram-mb=$((${NUM_NODES} * 60))  <del> --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota  </del> $(cat apiserver_flags) &> /var/log/kube-apiserver.log & # kube-contoller-manager now needs running kube-api server to actually start

Until is fixed (which won't happen before code-freeze is finished), kubemark-scale will be failing/timing-out and this is "working-as-intended" until then. It should be possible to switch off secrets in kubemark to make other problems debuggable. For now we would like to: switch off secrets in kubemark-scale leave them in all other kubemark suites
can you add a release-blocking or non-blocking label to this issue?
Is kubemark scale useful/realistic if it disables things that are required in actual deployments?
We use it for development, and we need a way to switch off particular parts.
- it's just a short term fix that would allow to fix/debug different problems at the same time (we already know that secrets are causing problems and we will be fixing it).
To clarify - I'm not going to leave that forever - as soon as we will have some fix for secrets in, we will enable it back in 5000-node kubemark. Currently because of secrets those tests doesn't bring any value.
Also - this one is already done.


Question: - resource - subresource - version <ins> - name: apiserver_response_sizes help: Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component. type: Histogram stabilityLevel: STABLE labels: - component - group - resource - scope - subresource - verb - version buckets: - 1000 - 10000 - 100000 - 1e+06 - 1e+07 - 1e+08 - 1e+09 </ins> - name: apiserver_storage_objects help: Number of stored objects at the time of last check split by kind. type: Gauge

I would like to propose the graduation of the following apiserver metrics from alpha to stable: With the exception of the metric these metrics have been stable for 2 years. The metric has been stable for 12 months.
/sig api-machinery /sig instrumentation
/assign /triage accepted
I'm okay with promoting all of these except I am hesitant about since it is relatively new. Can we omit that one?
I agree with the metric was patched a year ago as part of , so it would be best to wait a bit more until the majority of Kubernetes users catch up to the patched version and start using the new version of the metric in production.
A couple of renaming suggestions, since it is important to get it right before graduation: apiserverresponsesizes =apiserverresponsesizebytes aggregatorunavailableapiservice =aggregatorunavailableapiservices One concern regarding . It was very recently changed but the change was later reverted by In the end, the metric wasn't really changed, but it might raise the concern that some users may still need additional information from this metric so I am on the fence with this one. Side implementation note that we will need to keep in mind if we graduate is that the graduation should only affect the webhook admission metric and not the other, so we will need to pass the stability level to
Thank you and for the feedback. To clarify then, are the decisions to not graduate and final?
I'm actually okay with , the namespace addition was recent but it was a bad idea and was reverted. It has otherwise been in use for quite some time, so I'd be comfortable with promoting it.
I'm willing to make a PR for this one
+1 it wasn't a hard no from my side, just wanted to raise some concerns in case anyone had an objection.
/assign /unassign
Thank you for the feedback, and To summarize, we are in agreement to graduate all but the metric. Please correct me if I'm wrong.
/cc
since all the three metrics that are created with this function are graduating to stable, do you think I should still pass the to the function or should I just set the metrics created by this functions as stable by default? update: Seems like is designed to parse whatever's in front of . So putting a variable there (that is passed to the function) will cause the script to fail. Please correct me if I'm wrong.

update: Even setting the metrics to be stable by default will cause the script to fail and you'll get an error saying for the following line: It appears that the name of the stable metrics should be given to that object as a string and using the output of a function ( in this case) will cause the script to fail.
The PR is ready to be reviewed:


Question: state:             state, containerRuntime:  mockRuntimeService{}, containerMap:      containermap.NewContainerMap(), <del> activePods:        func() []*v1.Pod { return nil }, </del> podStatusProvider: mockPodStatusProvider{}, <ins> sourcesReady:      &sourcesReadyStub{}, activePods: func() []*v1.Pod { return []*v1.Pod{testCase.pod} }, </ins> } containers := append(

What happened: A container which had exclusive CPUs assigned to it (and an entry in /var/lib/kubelet/cpumanagerstate specifying those CPUs) was killed. When it came back, it was using the default cpuset, and its entry in /var/lib/kubelet/cpumanagerstate was gone. What you expected to happen: It should have been given exclusive CPUs. How to reproduce it (as minimally and precisely as possible): Create a pod/container with "guaranteed" QoS. Determine the exclusive CPUs allocated to the container. From the host, kill pid1 within the container. ("kill -9" works.) The container will restart within the existing pod, but it will be allocated cpus from the default cpuset (which will now include the CPUs that had been exclusively allocated to the container earlier). Anything else we need to know?: I some instrumentation to kubelet. What seems to be happening is this: 1) The container is killed (or dies). Kubelet then immediately creates a new container and calls PolicyStatic.Allocate() to allocate exclusive CPUs for it. This updates the entry in /var/lib/kubelet/cpumanagerstate (and the in-memory cache) which is mapped by podUID and container name, such that it contains the cpuset for the new container. 2) The InternalContainerLifecycle.PostStopContainer() code runs, which calls i.cpuManager.RemoveContainer(containerID). This uses the containerID of the exited container to look up the podUID/container-name tuple, then deletes the cpuset entry from /var/lib/kubelet/cpumanagerstate (and the in-memory cache). But the entry that was deleted was actually the one for the new container. 3) Kubelet calls the CPU manager's AddContainer() function, which calls m.state.GetCPUSetOrDefault(string(p.UID), c.Name) to get the cpuset based on the podUID and container name of the new container. The cpuset entry was erroneously deleted, so this returns the default cpuset, and the container which should have gotten exclusive CPUs ends up running on the default cpuset. An solution suggested by is to delay calling cpuManager.RemoveContainer() until the whole pod is deleted. This would preserve the CPU allocations.

An alternate solution also suggested by is to avoid calling cpuManager.RemoveContainer() from the container lifecycle code, but instead call removeStaleState() in the CpuManager.Allocate() function. Environment: Kubernetes version (use ): 1.18.1 Cloud provider or hardware configuration: x86, SMP OS (e.g: ): CentOS 7 Kernel (e.g. ): 3.10.0-1062.1.2.rt56.1025.el7
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Considering the following change based on discussion over Slack with
Sorry that bother you via this close issue, but I was tracing back to this issue via the PR. As per your statement above, if I understand correctly, this problem only happened when the container inside Pod got killed. but an simple Pod restarted (e.g. delete/re-create via kubectl delete pod command) should not trigger the problem, right? Or any container(inside Pod) or Pod restart with guaranteed Qos will trigger this issue?


Question: var extenders []algorithm.SchedulerExtender if len(policy.ExtenderConfigs) != 0 { ignoredExtendedResources := sets.NewString() <ins> var ignorableExtenders []algorithm.SchedulerExtender </ins> for ii := range policy.ExtenderConfigs { klog.V(2).Infof("Creating extender with config %+v", policy.ExtenderConfigs[ii]) extender, err := core.NewHTTPExtender(&policy.ExtenderConfigs[ii]) if err != nil { return nil, err } <del> extenders = append(extenders, extender) </del> <ins> if !extender.IsIgnorable() { extenders = append(extenders, extender) } else { ignorableExtenders = append(ignorableExtenders, extender) } </ins> for _, r := range policy.ExtenderConfigs[ii].ManagedResources { if r.IgnoredByScheduler { ignoredExtendedResources.Insert(string(r.Name)) } } } <ins> // place ignorable extenders to the tail of extenders extenders = append(extenders, ignorableExtenders...) </ins> predicates.RegisterPredicateMetadataProducerWithExtendedResourceOptions(ignoredExtendedResources) } // Providing HardPodAffinitySymmetricWeight in the policy config is the new and preferred way of providing the value.

What would you like to be : Put non-ignorable extenders ahead of ignorable extenders in genericScheduler#extenders Why is this needed: In genericScheduler#processPreemptionWithExtenders, we iterate over genericScheduler#extenders. If an ignorable extender gives error, the error is ignored and loop continues. It seems we should place non-ignorable extenders ahead of ignorable extenders so that the preemption can bail out early if a non-ignorable extender gives error.
/sig scheduling


Question: f.PodClient().DeleteSync(burstablePodName, &metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) f.PodClient().DeleteSync(bestEffortPodName, &metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) f.PodClientNS(kubeapi.NamespaceSystem).DeleteSyncInNamespace(criticalPodName, kubeapi.NamespaceSystem, &metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) <ins> err := f.ClientSet.SchedulingV1().PriorityClasses().Delete(systemCriticalPriorityName, &metav1.DeleteOptions{}) framework.ExpectNoError(err) </ins> // Log Events logPodEvents(f) logNodeEvents(f)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: if IsPodActive(p) { result = append(result, p) } else { <del> logger.V(4).Info("Ignoring inactive pod", "pod", klog.KObj(p), "phase", p.Status.Phase, "deletionTime", p.DeletionTimestamp) </del> <ins> logger.V(4).Info("Ignoring inactive pod", "pod", klog.KObj(p), "phase", p.Status.Phase, "deletionTime", klog.SafePtr(p.DeletionTimestamp)) </ins> } } return result

Job controller, and possibly other controllers keep logging panics from this line in : This happens during e2e tests, and I think it happens on production as well. Here is an example for the successful build: 2023-10-20T09:10:28.016158573Z stderr F I1020 09:10:28. 1 ] "Ignoring inactive pod" pod="ttlafterfinished-3394/rand-non-local-ghcvx" phase="Failed" deletionTime="<panic: runtime error: invalid memory address or nil pointer dereference" No panics during e2e test from this line in . Run e2e or integration tests for the job controller. No response <details</details<detailsReproducible on kind during e2e tests </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps FYI
Do you have any idea which variable is nil here?
oops, sorry, it's the deletionTimestamp
Is it actually a panic though? It seems that the logger is parsing a nil value as and proceeding. I think we could use and provide a default value. /priority important-longterm
Possibly this is just logger indeed. Still it is important imo because it makes looking for real panics in logs hard. This was my use case to understand another issue.
The logger sees an (= ) value which implements and calls it's method - in this case, ;l=527. That then panics. We've gone back and forth whether the logging call should try to call the method and recover from a panic or try to do some advance checks whether the value is perhaps nil. There could be implementations for pointers which produce some nice output for , so not calling it would be a loss.

Also, detecting nil values is surprisingly hard, if I remember correctly (the interface is non-nil, only the value hidden inside it is)), therefore currently the code simply calls it.
I can work of this /assign
how do you want to solve this? Besides some potential klog improvement, fixing to not crash for a nil pointer would be good. already does this.
Hi ! Sorry I didn't answer yesterday. I like the idea of doing the check at type level and we already have a precedent. Should we provide a default value as suggests? Tbh I'm not sure what would be a a proper default value here
When logging an object's , "is nil" is the right information to log when the pointer is nil. There is no default that can or should be used instead.
I've raised a PR with a simple fix. I've run the integration tests locally and it fixes the problem. Let me know if that's enough Thanks!
the changes in have been merged but not yet released (). what's the klog release cycle? It seems that there's no clear release schedule, right?
We usually do one release for each Kubernetes release, if there are changes.
However, in this case the next release this could go into is 1.30, right? This would be unfortunate because we wouldn't be able to fix 1.29 with the klog changes. If this is the case, I think we may need to do the quick if in ?
cc is there a chance to do another release of klog, so that we can use it in a patch release of k/k 1.29?
For 1.29, is a better fix than rushing a new klog release. But is this critical enough to ask the release team for an exception?
It only affects developers who bump up the verbosity to = 4.
So we can keep 1.29 as is, or do the quick fix in . I would suggest to do something about it, because it makes our log artifacts not so useful to detect real panics - a simple grep does not work anymore. do you think it makes sense to patch 1.29? Oh, yes, probably it is not critical enough, I keep forgetting about the = 4 level.
It's definitely not critical.

It's not even a real panic... it's just a log line that says "panic"
so better wait for the and then make the change? or add meanwhile?
I would wait, given that the cherry-pick is not justified


Question: case statInfo.Mode().IsRegular(): pod, err := s.extractFromFile(path) if err != nil { <del> glog.Errorf("Can't process manifest file %q: %v", path, err) </del> <ins> if !os.IsNotExist(err) { glog.Errorf("Can't process manifest file %q: %v", path, err) } </ins> } else { pods = append(pods, pod) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: sometimes use vi edit /etc/kubernetes/manifests/kube-, apiserver will be kill, but not restart again; but sometimes is ok, kube-apiserver will restart.(test 10 times, and 2 times problem found) but if I move this file out of /etc/kubernetes/manifests/, wait for moment, and then move back, apiserver restart, problem fix. What you expected to happen: edit /etc/kubernetes/manifests/kube-, apiserver auto-update How to reproduce it (as minimally and precisely as possible): a k8s cluster. kube- by vi editor. apiserver status. Anything else we need to know?: any thing you want to know, please ping me. Environment: Kubernetes version (use ):Cloud provider or hardware configuration:none OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm Others:none
/sig node
could you help to fix this problem?
You mean changing file in folder ? Seems did not catch your changes. Any logs from ?
yes, and I have try again, got this:
but why is 4913, not the file kube- I have try several times, and got the same error message. by the way, following code, I arrive here: (), seem here to watch the file change.
Thanks for finding this bug. I will take a look.
after I add glog.Errorf("Be careful, I am going to check: %v", e.Name) to print log, I found this: has read the right file /etc/kubernetes/kube- must parse to pod success, because it knows the mirror older kube-apiserver is outdated add in ():
Right. was deleted first. That's why so many occurred afterwards. And the new pod for cannot be created as well. Seems we have to make a special treatment for , but this will be kind of ugly. Any suggestions?

relatived to but found in kubernetes v1.8.3 further more, following the code, I have found that create mirror pod failed, because kubelet can't connect to apiserver, and the code already filter the temp file .kube- So, the problem must be why kubelet can't start kube-apiserver static pod. But I can't find out where to create the static pod in code, Could you help?
is in (), I will continue to find out why kube-apiserver not start.
Good job! Thanks for the update. I prefer to move the discussion to .
It's a good idea, discussion in .
/close Move discussion to .


Question: exec := exec.New() out, err := exec.Command(nsenterPath, args...).CombinedOutput() if err != nil { <del> glog.V(2).Infof("Failed findmnt command: %v", err) </del> <ins> glog.V(2).Infof("Failed findmnt command for path %s: %v", file, err) </ins> // Different operating systems behave differently for paths which are not mount points. // On older versions (e.g. 2.20.1) we'd get error, on newer ones (e.g. 2.26.2) we'd get "/". // It's safer to assume that it's not a mount point.

Currently, there many error messages in that do not contain any context. This means that these messages are basically meaningless in real systems where many pods may be on a node. Examples: Every error message in this file should identify the path that the message is in relation to.
Can you handle this, since you most recently touched in ?


Question: } loadBalancerName := cloudprovider.GetLoadBalancerName(service) <del> pool, err := gce.GetTargetPool(loadBalancerName, gce.region) if err != nil { return err } existing := sets.NewString() for _, instance := range pool.Instances { existing.Insert(hostURLToComparablePath(instance)) } return gce.updateTargetPool(loadBalancerName, existing, hosts) </del> <ins> return gce.updateTargetPool(loadBalancerName, hosts) </ins> } // ensureExternalLoadBalancerDeleted is the external implementation of LoadBalancer.EnsureLoadBalancerDeleted

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area cloudprovider What happened: The cloud provider interface that should . The GCE provider does this for but not for . Every other in-tree provider implements by fully updating the load balancer including updated backend nodes: , , , And the following out-of-tree providers do as well and Oracle (not oss yet). What you expected to happen: I expect that all cloud providers implement by creating or fully updating the load balancer if it already exists. Related discussion: /cc
/assign Make sense to make implementation in GCE consistent with others for the ease of up-coming service controller enhancement. This sounds like a 1.10 task.
is this task blocking you on another attempt at refactoring the service controller sync logic?
yup


Question: scoreWithNormalizePlugin1         = "score-with-normalize-plugin-1" scoreWithNormalizePlugin2         = "score-with-normalize-plugin-2" scorePlugin1                      = "score-plugin-1" <ins> scorePlugin2                      = "score-plugin-2" </ins> pluginNotImplementingScore        = "plugin-not-implementing-score" preFilterPluginName               = "prefilter-plugin" preFilterWithExtensionsPluginName = "prefilter-with-extensions-plugin"

Scheduler's default plugin returns an invalid score that is not in the range of [0, 100], resulting in no Pods being scheduled: After investigation, I found that there was a bug in the code that built the framework when the scheduler started. When using the plugins declared in the regular extension point to override the plugins declared in the MultiPoint, the traversed variable will be modified in the following loop: This may cause a same scoring plugin (e.g. ) to be loaded multiple times: Further, this results in a node being processed multiple times by a plugin's (e.g. 's) method, resulting in an invalid score that is not in the range of [0, 100]: If the scheduler's MultiPoint feature is used (either through the default plugins or through manual configuration), and the configuration (e.g. the scoring weight) of a certain plugin is overridden in a regular extension point, the plugin will only be loaded once in this extension point, and no illegal scores will be returned. An example KubeSchedulerConfiguration: No response K8s v1.24.15 or higher None <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Can I work on this?
this is being worked on. see
I see, Is there any other task that you are aware of and I can work on? Thanks.


Question: "k8s.io/kubectl/pkg/util/templates" ) <ins> const ( MsgNotLabeled = "not labeled" MsgLabeled    = "labeled" MsgUnLabeled  = "unlabeled" ) </ins> // LabelOptions have the data required to perform the label operation type LabelOptions struct { // Filename options

Currently, the cmd output msg is not distinguished between label and unlabel (using suffix ), both of them response . To distinguish between label and unlabel msg: , just like .
/sig cli
/triage accepted


Question: q := NewPriorityQueue(nil) q.Add(&highPriNominatedPod) q.AddUnschedulableIfNotPresent(&highPriNominatedPod, q.SchedulingCycle()) // Must not add anything. <del> q.AddUnschedulableIfNotPresent(&medPriorityPod, q.SchedulingCycle())      // This should go to activeQ. </del> q.AddUnschedulableIfNotPresent(&unschedulablePod, q.SchedulingCycle()) expectedNominatedPods := &nominatedPodMap{ nominatedPodToNode: map[types.UID]string{ <del> medPriorityPod.UID:      "node1", </del> unschedulablePod.UID:    "node1", highPriNominatedPod.UID: "node1", }, nominatedPods: map[string][]*v1.Pod{ <del> "node1": {&highPriNominatedPod, &medPriorityPod, &unschedulablePod}, </del> <ins> "node1": {&highPriNominatedPod, &unschedulablePod}, </ins> }, } if !reflect.DeepEqual(q.nominatedPods, expectedNominatedPods) {

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: "reflect" "sync" "testing" <ins> "time" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scheduler places unschedulable pods in "unschedulabe" queue and retries them only when certain events happen that could potentially make them schedulable. This logic works well in almost all scenarios, but inevitable race condition in large distributed systems, could potentially cause some events to be seen before pods are to the unschedulable queue. If this happens, pods may be left in the unschedulable queue and not be retried. Such scenarios should be rare and even if they occur, usually there are other events that trigger a retry and cover them. However, if such scenarios happen in smaller and low churn clusters, other events may not be seen for a while and pods may be stuck in the unschedulable queue for a long time. The scheduling queue () has a method that is triggered every second to move backed off pods to the active queue. We should add another go routine that is triggered every 30 seconds and moves pods whose last schedule time is older than 1 minute from "unschedulble" queue to the active queue. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug I don't know if this should be marked as a bug. I don't have any report of such issue. It is more of a precaution to prevent issues. /sig scheduling
I am worried if this will lead to continuous repeated scheduling failures?
What is the problem with repeated scheduling failures? The scheduler had a FIFO until recently. It used to keep retrying unschedulable pods all the time.
can I take this?
Sure, thanks!
GitHub didn't allow me to assign the following users: denkensk. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It increases the schedule possibility of unschedulble pods and the chances of scheduling failures. Should we also consider latter?


Question: nodes                   []string kubeClient              client.Interface kubeletClient           client.KubeletClient <ins> recorder                record.EventRecorder </ins> registerRetryCount      int podEvictionTimeout      time.Duration deletingPodsRateLimiter util.RateLimiter

/cc
Is this from head? I've never seen this before, maybe changed anything recently. Anyway, the code in question will be deleted by
I saw this in the Travis run for an unrelated PR (Kubelet test refactoring). If it's going away, then that's ho we'll fix it :)
Yup. It'll go away. I'll assign it to myself - I'll run test few times after to make sure it's gone.
another example: https://travis-
Just hit this again: https://travis-
I was hoping we could finish getting rid of this code this week, which is why I had been ignoring this. But we ran out of time and Poland has a holiday on Monday, so I guess we need to fix this race. I'll take a look.
Thanks. This is becoming a real nuisance that is masking other failures.
To repro I assume I just type the commands from Travis, e.g. the shell stuff in https://travis- ?
Never mind, I got it. Though the first time I ran it all the tests passed.
I don't think the failure is deterministic, but yes.
This is super annoying, it happens way too much of the time...
Sigh, I was hoping we can get rid of the code this week. I think the problem comes from someone changes the event recording. I'll send a PR
It seems to be basically always failing now, except when some other flake fails first.
It would be fine with me if we just temporarily disabled event generation there.
If I'm reading the stack traces correctly, what's going on is that NewNodeController [] calls StartRecordingToSink() [] which calls StartEventWatcher() which calls eventHandler() which calls StartRecordingToSink() which calls sink.Create() which is Create() in , which appends to c.Fake.Actions. At the same time, the last line of the test tries to print "item.fakeNodeHandler.Actions" and these are presumably the same variable. I'm not sure why these are running at the same time, though.
+1 to Braians idea to temporary disable event generation.
Re-titled issue.
Re-titled issue.


Question: } if docsForTypes != nil && len(docsForTypes) > 0 { <del> if err := pkg_runtime.WriteSwaggerDocFunc(docsForTypes, funcOut); err != nil { </del> <ins> if err := kruntime.WriteSwaggerDocFunc(docsForTypes, funcOut); err != nil { </ins> fmt.Fprintf(os.Stderr, "Error when writing swagger documentation functions: %sn", err) os.Exit(-1) }

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package app import ( "testing" "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/discovery" fakediscovery "k8s.io/client-go/discovery/fake" "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes" fakeclientset "k8s.io/client-go/kubernetes/fake" restclient "k8s.io/client-go/rest" ) // TestClientBuilder inherits ClientBuilder and can accept a given fake clientset. type TestClientBuilder struct { clientset clientset.Interface } func (TestClientBuilder) Config(name string) (*restclient.Config, error)  { return nil, nil } func (TestClientBuilder) ConfigOrDie(name string) *restclient.Config      { return nil } func (TestClientBuilder) Client(name string) (clientset.Interface, error) { return nil, nil } func (m TestClientBuilder) ClientOrDie(name string) clientset.Interface { return m.clientset } // FakeDiscoveryWithError inherits DiscoveryInterface(via FakeDiscovery) with some methods accepting testing data. type FakeDiscoveryWithError struct { fakediscovery.FakeDiscovery PossibleResources []*metav1.APIResourceList Err               error } func (d FakeDiscoveryWithError) ServerPreferredNamespacedResources() ([]*metav1.APIResourceList, error) { return d.PossibleResources, d.Err } // FakeDiscoveryWithError inherits Clientset(via FakeClientset) with overridden Discovery method. type FakeClientSet struct { fakeclientset.Clientset DiscoveryObj *FakeDiscoveryWithError } func (c *FakeClientSet) Discovery() discovery.DiscoveryInterface { return c.DiscoveryObj } // Create a fake Clientset with its Discovery method overridden. func NewFakeClientset(fakeDiscovery FakeDiscoveryWithError) *FakeClientSet { cs := &FakeClientSet{} cs.DiscoveryObj = &fakeDiscovery return cs } func TestStartResourceQuotaController_DiscoveryError(t *testing.T) { tcs := map[string]struct { discoveryError    error expectedErr       bool possibleResources []*metav1.APIResourceList }{ "No Discovery Error": { discoveryError:    nil, possibleResources: nil, expectedErr:       false, }, "Discovery Calls Partially Failed": { discoveryError: new(discovery.ErrGroupDiscoveryFailed), possibleResources: []*metav1.APIResourceList{ { GroupVersion: "create/v1", APIResources: []metav1.APIResource{ { Name:       "jobs", Verbs:      []string{"create", "list", "watch", "delete"}, ShortNames: []string{"jz"}, Categories: []string{"all"}, }, }, }, }, expectedErr: false, }, } for name, test := range tcs { testDiscovery := FakeDiscoveryWithError{Err: test.discoveryError, PossibleResources: test.possibleResources} testClientset := NewFakeClientset(testDiscovery) testClientBuilder := TestClientBuilder{clientset: testClientset} ctx := ControllerContext{ ClientBuilder:    testClientBuilder, InformerFactory:  informers.NewSharedInformerFactoryWithOptions(testClientset, time.Duration(1)), InformersStarted: make(chan struct{}), } _, _, err := startResourceQuotaController(ctx) if test.expectedErr != (err != nil) { t.Errorf("test failed for use case: %v", name) } } } </ins>

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: t.Errorf("did not set resources: %s", buf.String()) } } <ins> func TestSetMultiResourcesLimitsLocal(t *testing.T) { f, tf, codec, ns := cmdtesting.NewAPIFactory() tf.Client = &fake.RESTClient{ GroupVersion:         legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion, NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { t.Fatalf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, nil }), } tf.Namespace = "test" tf.ClientConfig = &restclient.Config{ContentConfig: restclient.ContentConfig{GroupVersion: &legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion}} buf := bytes.NewBuffer([]byte{}) cmd := NewCmdResources(f, buf, buf) cmd.SetOutput(buf) cmd.Flags().Set("output", "name") cmd.Flags().Set("local", "true") mapper, typer := f.Object() tf.Printer = &printers.NamePrinter{Decoders: []runtime.Decoder{codec}, Typer: typer, Mapper: mapper} opts := ResourcesOptions{FilenameOptions: resource.FilenameOptions{ Filenames: []string{"../../../../test/fixtures/pkg/kubectl/cmd/set/multi-resource-yaml.yaml"}}, Out:               buf, Local:             true, Limits:            "cpu=200m,memory=512Mi", Requests:          "cpu=200m,memory=512Mi", ContainerSelector: "*"} err := opts.Complete(f, cmd, []string{}) if err == nil { err = opts.Validate() } if err == nil { err = opts.Run() } if err != nil { t.Fatalf("unexpected error: %v", err) } expectedOut := "replicationcontrollers/first-rcnreplicationcontrollers/second-rcn" if buf.String() != expectedOut { t.Errorf("expected out:n%snbut got:n%s", expectedOut, buf.String()) } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign


Question: }, sz, nil } <ins> func (p *glusterfsVolumeProvisioner) createEndpointService(namespace string, epServiceName string, hostips []string, pvcname string) (endpoint *v1.Endpoints, service *v1.Service, err error) { addrlist := make([]v1.EndpointAddress, len(hostips)) for i, v := range hostips { addrlist[i].IP = v } endpoint = &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{ Namespace: namespace, Name:      epServiceName, Labels: map[string]string{ "gluster.kubernetes.io/provisioned-for-pvc": pvcname, }, }, Subsets: []v1.EndpointSubset{{ Addresses: addrlist, Ports:     []v1.EndpointPort{{Port: 1, Protocol: "TCP"}}, }}, } kubeClient := p.plugin.host.GetKubeClient() if kubeClient == nil { return nil, nil, fmt.Errorf("glusterfs: failed to get kube client when creating endpoint service") } _, err = kubeClient.Core().Endpoints(namespace).Create(endpoint) if err != nil && errors.IsAlreadyExists(err) { glog.V(1).Infof("glusterfs: endpoint [%s] already exist in namespace [%s]", endpoint, namespace) err = nil } if err != nil { glog.Errorf("glusterfs: failed to create endpoint: %v", err) return nil, nil, fmt.Errorf("error creating endpoint: %v", err) } service = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      epServiceName, Namespace: namespace, Labels: map[string]string{ "gluster.kubernetes.io/provisioned-for-pvc": pvcname, }, }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{ {Protocol: "TCP", Port: 1}}}} _, err = kubeClient.Core().Services(namespace).Create(service) if err != nil && errors.IsAlreadyExists(err) { glog.V(1).Infof("glusterfs: service [%s] already exist in namespace [%s]", service, namespace) err = nil } if err != nil { glog.Errorf("glusterfs: failed to create service: %v", err) return nil, nil, fmt.Errorf("error creating service: %v", err) } return endpoint, service, nil } func (d *glusterfsVolumeDeleter) deleteEndpointService(namespace string, epServiceName string) (err error) { kubeClient := d.plugin.host.GetKubeClient() if kubeClient == nil { return fmt.Errorf("glusterfs: failed to get kube client when deleting endpoint service") } err = kubeClient.Core().Services(namespace).Delete(epServiceName, nil) if err != nil { glog.Errorf("glusterfs: error deleting service %s/%s: %v", namespace, epServiceName, err) return fmt.Errorf("error deleting service %s/%s: %v", namespace, epServiceName, err) } glog.V(1).Infof("glusterfs: service/endpoint %s/%s deleted successfully", namespace, epServiceName) return nil } </ins> // parseSecret finds a given Secret instance and reads user password from it. func parseSecret(namespace, secretName string, kubeClient clientset.Interface) (string, error) { secretMap, err := volutil.GetSecretForPV(namespace, secretName, glusterfsPluginName, kubeClient)

several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.


Question: # Enable features ENABLE_FEATURE_GATES="" <ins> VERSION_OVERRIDE="" if [[ "${CUSTOM_VERSION_SUFFIX:-}" != "" ]]; then VERSION_OVERRIDE="--version=$("${THIS_PLATFORM_BIN}/kube-apiserver" --version | awk '{print $2}')${CUSTOM_VERSION_SUFFIX:-}" fi </ins> "${THIS_PLATFORM_BIN}/kube-apiserver"  <ins> ${VERSION_OVERRIDE:+"${VERSION_OVERRIDE}"}  </ins> --bind-address="127.0.0.1"  --authorization-mode="${AUTHORIZATION_MODE}"  --secure-port="${SECURE_API_PORT}"

Starting in 1.31, --version no longer works to modify the build ID of the running components. /kind regression /priority important-soon Running kube-apiserver v1.31.0 with ignores the modified build ID and reports In 1.28, 1.29, and 1.30 (since ), this allowed modifying the build ID of the running server. /sig api-machinery /assign Run v1.31.0 with No response 31.0 n/a <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this broke in DefaultBuildEffectiveVersion() is grabbed in NewAPIServerCommand(), before the flag is parsed, so the GitVersion propagated into the snapshot effectiveVersion doesn't honor
cc


Question: }) podUID := string(pod.UID) ginkgo.By("checking if the expected hugetlb settings were applied") <del> verifyPod := makePodToVerifyHugePages("pod"+podUID, resource.MustParse("50Mi")) </del> <ins> verifyPod := makePodToVerifyHugePages("pod"+podUID, resource.MustParse("6Mi")) </ins> f.PodClient().Create(verifyPod) err := e2epod.WaitForPodSuccessInNamespace(f.ClientSet, verifyPod.Name, f.Namespace.Name) framework.ExpectNoError(err)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: -tests Since when has it been failing: Unsure Testgrid link: Reason for failure: Unable to pre allocate 50x2MiB huge pages, probably due to memory pressure caused by kernel level caching. If requesting an substantial amount of huge page memory (like the tests), it is recommended to to it as early after boot as possible, or by using the kernel cmd line. Eg. this one: Anything else we need to know: will fix this issue, but in awaiting for that, we can just decrease the page count in the tests.
/sig node /cc derekwaynecarr
I think the reason of railure should be because a single hugepage of x86 is 2MiB as and a single page is 4KiB, not 2KiB.
Yes, is correct. Thanks for catching the typo :smile:
Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. This is probably because the kernel is caching a lot of stuff, making it hard for the kernel to fix it. Ill do some testing with dropping the caches (no user facing stuff will happen, only kernel level) before trying to pre allocate huge pages. /reopen
Reopened this issue. <detailsIn response to : Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> // FrameworkCapturer is used for registering a notify function in building framework. type FrameworkCapturer func(schedulerapi.KubeSchedulerProfile) // WithBuildFrameworkCapturer sets a notify function for getting buildFramework details. func WithBuildFrameworkCapturer(fc FrameworkCapturer) Option { return func(o *schedulerOptions) { o.frameworkCapturer = fc } } </ins> var defaultSchedulerOptions = schedulerOptions{ profiles: []schedulerapi.KubeSchedulerProfile{ // Profiles' default plugins are set from the algorithm provider.

has compatibility tests for Policy v1. We are lacking similar tests for other options of , such as: [x] : This can be right away to test that default plugins can be disabled, changed order, etc. [x] : This one is trickier, as it requires a new method in to expose all configurations and a new method in so that they can say which args they used. /sig scheduling /help /kind cleanup
/assign
you can work on the first one. I'll have a PR soon that should clarify how the second one should be done.
I have sent a pr for the first one. And I'm working on the second one, I think it's better to keep 2 PRs for this issue? If so, PTAL at the first one, thanks.
Yes, 2 PRs is ideal. I'll review later today.
I have one small question about do I need to mock some fake plugins for testing , or just testing the default plugins?
You should use the existing plugins, because these are compatibility tests.
Okay
Actually, I think adding a new method such as to interface is a little redundant. In fact, most plugins don't need any argument for building them, so we are talking about adding a new method that needs to be implemented by all of the existing and coming plugins. And users who write out-of-tree plugins also hardly have the need for using this method. IMO, it would be more meaningful to move up to the interface. Then by this method, we can simply implement for default plugins who actually need args. WDYT?
should remain unexported because it provides pointers. However, I agree that might be too much to ask for all plugins. Instead, we can add a new interface and then plugins can opt-int. Not sure if we should follow this extension pattern Any thoughts
On the other hand, simply returning doesn't seem like much.
Sure, then I will add like what we want before(add this method to interface), and if there is any better idea later, please let me know.
I think we should just add it to the plugins that actually have args, this will require custom code in the test, but I think it is better than having all plugins implement a function means something to only 1% of them.
Thanks, but now there is a problem that we can't get the registered plugins with current API.

So, I'm not sure if we can just implement the method in with some reflect code for checking if one plugin has implement method? This could help us with keeping the interface clean. cc
you don't need to check if it implements BuildArgs, since this is for testing, we can hardcode which plugins should have their args verified.
The test will not be pretty, but I think it is worth the tradeoff.
Do you mean that we don't test the default plugins, but define some custom plugins in the test file? And we don't need the ListPluginArgs method anymore, am I right?
and I discussed this offline and arrived to this conclusion: For the compatiblity test, we can mock ( in ) to capture the arguments the configuration that is being passed. This can also be used by with We want another test for that checks the build args for the in-tree plugins accessing the internal pointers. These 2 tests should cover the whole stack. Sorry for the back-and-forth. Sometimes it's useful to see a full PR to realize that the initial idea wasn't good enough. Thanks a lot for it.
Sure, that sounds much more clear, thanks. :D
One more question, the is under package , and the compatibility test is under package . So for mocking , we need to change to a function variable in , and add a method for it, right It will look like: Maybe we can add something like to , but that's the implementation details. I want to make sure if the above design is proper.
Or rather, you could add something like: . Then you can call the capturer from if there is one defined.
Got your message in slack, let me reply here. sounds good, and do I need to revert the change in ? I think is not necessary now.
I think they will still be useful for the other test:
Hello are you still working on this?
Ah, I'm so sorry that I'm so busy in the last few weeks, but I will send prs fot this issue in this weekend. Sorry for my delay again.


Question: runOutput := framework.NewKubectlCommand(nsFlag, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "cat && echo 'stdin closed'"). WithStdinData("abcd1234"). ExecOrDie() <del> Expect(runOutput).To(ContainSubstring("abcd1234")) Expect(runOutput).To(ContainSubstring("stdin closed")) </del> <ins> g := func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err := polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test", 1*time.Minute, g) Expect(err).To(BeNil()) // NOTE: we cannot guarantee our output showed up in the container logs before stdin was closed, so we have // to loop test. err = wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { if !framework.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, 1*time.Second) { framework.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test") } logOutput := framework.RunKubectlOrDie(nsFlag, "logs", runTestPod.Name) Expect(runOutput).To(ContainSubstring("abcd1234")) Expect(runOutput).To(ContainSubstring("stdin closed")) return strings.Contains(logOutput, "abcd1234"), nil }) Expect(err).To(BeNil()) </ins> Expect(c.BatchV1().Jobs(ns).Delete("run-test", nil)).To(BeNil()) By("executing a command with run and attach without stdin")

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: } // However, "empty" preFilterState is legit which tolerates every toSchedule Pod. <del> if len(s.TpPairToMatchNum) == 0 || len(s.Constraints) == 0 { </del> <ins> if len(s.Constraints) == 0 { </ins> return nil }

What happened: I was testing out the pod topology spread functionality in a local k8s cluster. I noticed that when I set the constraint to be pods are still successfully scheduled. What you expected to happen: Reading the documentation on , I expected pods not to be scheduled since the node should be "bypassed due to the absence of label". Did I miss something here? How to reproduce it (as minimally and precisely as possible): Create a pod with topologySpreadConstraints[0].topologyKey that does not match to any node labels, and set to . Anything else we need to know?: Thank you Environment: Kubernetes version (use ):Cloud provider or hardware configuration: It's a kind cluster created with image . OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
This sounds like a bug, I will look into it. /assign


Question: } func (gce *GCECloud) firewallNeedsUpdate(name, serviceName, region, ipAddress string, ports []v1.ServicePort, sourceRanges netsets.IPNet) (exists bool, needsUpdate bool, err error) { <del> fw, err := gce.service.Firewalls.Get(gce.NetworkProjectID(), makeFirewallName(name)).Do() </del> <ins> fw, err := gce.service.Firewalls.Get(gce.NetworkProjectID(), MakeFirewallName(name)).Do() </ins> if err != nil { if isHTTPErrorCode(err, http.StatusNotFound) { return false, true, nil

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> FROM BASEIMAGE </del> <ins> FROM --platform=linux/ARCH BASEIMAGE </ins> COPY ginkgo /usr/local/bin/ COPY e2e.test /usr/local/bin/

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: produces an error: What you expected to happen: Conformance tests should be launched. Examining image contents indicates executables are of the correct architecture. How to reproduce it (as minimally and precisely as possible): From aarch64 machine: Anything else we need to know?: Works with v1.17.4: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Architecture /sig Testing
/sig release /remove-sig architecture
any thoughts on this issue at all? We use this to test K3d and are blocked on releasing anything in the 1.18 line because of it.
I don't have expertise in the image release pipeline, I'd suggest reaching out to
Hi Did you run this test in an arm64 virtual machine? If so, I have fixed this issue in cadvisor side. This issue is caused by cadvisor. Currently, the version of cadvisor in kubernetes is v0.36.1-0.-. There is a defect in this version, it gets the wrong machine info and reports num_cores=0 on Arm64 guest. I have delivered the PR into cadvisor in 22 days ago. And this patch was merged in 10 days ago. Please see my pr as reference: Currently, I'm waiting a new tag of cadvisor, so that I can use this new tag in kubernetes and fix this issue in k8s side. Please see my issue: Thanks.
Currently we only have one Arm64 conformance test is alive, this conformance test was run on 2 AWS arm64 instances https://k8s- All other VM-based Arm64 conformance tests are down.
/area conformance
Thanks for taking a look just to be sure it is not a VM issue I fired up an a1-metal arm64 aws instance.

Please note I am using sonobuoy for running tests rather than kubetest, I tried to find if kubetest is using the conformance image for testing and from a quick check maybe not. The error is not in the conformance test itself, but launching the conformance test image on arm64. The is . Tried again with with 1.18.2 and got the same error: still works ok for the v1.17 branch: It looks like the build chain for images has changed: So there are now six less layers of the conformance image for the 1.18 branch. Please also note how the arm64 images show amd64 as the architecture for both branches. This has been problematic for trying to air-gap arm64 properly and is a systemic issue for many of the projects on gcr (and elsewhere).
Hi I see. I'll check it. Thanks.
/reopen This issue is still unfixed as of today. here is a reproducer that helps doing the reverse test on a x8664 machine: So this is an image where the base os is actually arm64. starting with v1.18.0 and still as of today (v1.18.5-rc.0 and v1.19.0-beta.2) the uploaded -arm64 image is actually an amd64 image. There must be a typo somewhere. testable this way: This is supposed to generate the 'exec file error', but it doesn't, which indicates that it is actually an x8664 image (amd64).
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen x8664 # podman pull # podman run 66652b9ed7f16a9bc848038ddb00159f3e2d22d8c2bb1ba050913309daf101c9 cat /etc/os-release exec user process caused "exec format error" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC the fix was reverted due to not working in the release pipeline. I did say that would hurt :upsidedownface:
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten This is still blocking conformance testing on arm64, so I hope it can be considered open.
Looks like to me this is a very complicated issue which is related with GCB, Moby Buildkit, it's not easy for me to reproduce and some trouble shooting by myself since I am not able to access GCB. Just wanna know is there any follow up to make this PR merge again? , we do have some needs to run the conformance testing on ARM64 .
SIG testing does not use this image (we prefer not to run the test runner as a pod given our clusters may be unreliable since we're testing changes and we want to avoid e.g. the pod being evicted =data loss), however this is currently used by the CNCF conformance program (which is CNCF / SIG arch) and sonobuoy (which is a vmware project). A bunch of our images are shipping with buildkit / buildx currently at HEAD, something like number could probably be brought back if someone wanted to work on this. Currently I'm not going to have time myself.
/cc
cc for arm64 image :(
xref /assign
It looks like that the issue is not in k/release, since we only build/stage the images via . Then we push the images and annotate them to their corresponding architecture.

Looking at the staged artifacts, for example for v1.20.1 arm64: Then we can see that also kube-proxy seems to have this issue: I assume that the fix is somewhere in the build scripts, where I'm now looking into. Beside that, all other architectures (arm, ppc64le, ...) are affected by this bug as well (for the same images only). Edit: Looks like the distroless images are fine, whereas the debian based images (proxy / conformance) are pointing to the wrong architecture. Conformance seems to always use , which might be the issue here. Edit: Done with my experiments. will change the already open PR according to our findings from the slack conversation (), whereas we have to change the used base images for kube-proxy to match the corresponding target architecture (like using for )