Question: klog.V(2).Info("DetectLocalMode: '", string(detectLocalMode), "'") <ins> primaryProtocol := utiliptables.ProtocolIPv4 if utilsnet.IsIPv6(nodeIP) { primaryProtocol = utiliptables.ProtocolIPv6 } iptInterface = utiliptables.New(execer, primaryProtocol) var ipt [2]utiliptables.Interface dualStack := utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) && proxyMode != proxyModeUserspace if dualStack { // Create iptables handlers for both families, one is already created // Always ordered as IPv4, IPv6 if primaryProtocol == utiliptables.ProtocolIPv4 { ipt[0] = iptInterface ipt[1] = utiliptables.New(execer, utiliptables.ProtocolIPv6) // Just because the feature gate is enabled doesn't mean the node // actually supports dual-stack if _, err := ipt[1].ChainExists(utiliptables.TableNAT, utiliptables.ChainPostrouting); err != nil { klog.Warningf("No iptables support for IPv6: %v", err) dualStack = false } } else { ipt[0] = utiliptables.New(execer, utiliptables.ProtocolIPv4) ipt[1] = iptInterface } } if dualStack { klog.V(0).Infof("kube-proxy running in dual-stack mode, %s-primary", iptInterface.Protocol()) } else { klog.V(0).Infof("kube-proxy running in single-stack %s mode", iptInterface.Protocol()) } </ins> if proxyMode == proxyModeIPTables { klog.V(0).Info("Using iptables Proxier.") if config.IPTables.MasqueradeBit == nil {

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: request: &http.Request{ Method: "PATCH", URL: &url.URL{ <del> RawPath:  "/apis/batch/v1/namespaces/foo/pods/status", </del> <ins> RawPath:  "/apis/batch/v1/namespaces/foo/jobs/bar/status", </ins> RawQuery: "dryRun=All", }, }, requestInfo: &request.RequestInfo{ <ins> Verb:              "patch", </ins> APIGroup:          "batch", APIVersion:        "v1", Resource:          "jobs", <ins> Name:              "bar", </ins> Subresource:       "status", IsResourceRequest: true, },

I've observed, that some metrics has and labels. (create) requests are "single resource", therefor they should be classified as . Def: customresourcehandler uses , which is based on the RequstInfo only: the RequestInfo is based only on the requests: method and url example: e.g: Since this RequestInfo has Name and Namespace empty, it is classified as . In a similar way it could be explained why: There's a mismatch in scope classification between built-in and custom resources: The key difference - the check if either "name" was provided OR it is a : Where the metrics.CleanScope ONLY checks, if name is not-empty. Tested: = v1.20 There were other scope related issues <1.20 and the distinction is not obvious. <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: exit 1 fi <ins> # Exit on error set -e </ins> # Use testing config export KUBE_CONFIG_FILE="config-test.sh" export KUBE_REPO_ROOT="$(dirname $0)/.."

It shouldn't: But my shell-fu is weak, maybe there's something wrong with that?
I think it's missing some quotes. I'll make a PR.


Question: RELEASE_TAR_PATH=$(dirname ${RELEASE_TAR_LOCATION}) echo "[INFO] Uploading ${KUBERNETES_RELEASE_TAR}" <del> swift upload kubernetes ${RELEASE_TAR_PATH}/${KUBERNETES_RELEASE_TAR}  </del> <ins> swift upload ${SWIFT_OBJECT_STORE} ${RELEASE_TAR_PATH}/${KUBERNETES_RELEASE_TAR}  </ins> --object-name kubernetes-server.tar.gz echo "[INFO] Uploading kubernetes-salt.tar.gz" <del> swift upload kubernetes ${RELEASE_TAR_PATH}/kubernetes-salt.tar.gz  </del> <ins> swift upload ${SWIFT_OBJECT_STORE} ${RELEASE_TAR_PATH}/kubernetes-salt.tar.gz  </ins> --object-name kubernetes-salt.tar.gz }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently swift object store has a hard-coded name: . E.g., What you expected to happen: Make swift object store parameterizable. E.g., How to reproduce it (as minimally and precisely as possible): This defaults to the hard-coded swift object store. Anything else we need to know?: Here is the patch file to address this issue: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., /sig scalability for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig openstack /sig storage /remove-sig api-machinery
cc
Quick question, if you have the diff already, why not send a PR?
I noticed the contributor documentation is extensive (b/c k8s is such a big project, this is understandable) and I wasn't sure how if I was following the rules. I can send the PR, if you would like. This is no problem. Let me know :-)
I created a PR based on the diff in your issue. Please let me know if you'd like me to abandon it so that you can have credit for the work.
Here you go:
hey can you please reopen this?
I don't think I have enough privileges to reopen. It's probably best to create a new issue and reference this one.
Ah. thanks
/assign
/reopen
Looks like the bot commands worked! ()
/close


Question: func (lb *LoadBalancerRR) resetService(svcPort proxy.ServicePortName) { // If the service is still around, reset but don't delete. <del> if state, ok := lb.services[svcPort]; ok { </del> <ins> if state, ok := lb.services[svcPort]; ok && state != nil { </ins> if len(state.endpoints) > 0 { klog.V(2).InfoS("LoadBalancerRR: Removing endpoints service", "servicePortName", svcPort) state.endpoints = []string{}

As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted


Question: // of having to do conversions with IP addresses. // Don't allocate the network's ".0" address. ipFirst := prefix.Masked().Addr().Next() <del> if err != nil { return nil, err } </del> // Use the broadcast address as last address for IPv6 ipLast, err := broadcastAddress(prefix) if err != nil {

The err is already checked on line 84, so it will always be nil On line 107, the check is redundant
That is indeed redundant. It seems like someone forgot to remove after a refactor but It was this way in
Feel free to send a pull request by following Thanks.
Thanks! I am wondering whether it is necessary to check ip address returned by calling or is As in But I don't find the overflowed cases yet.
/sig network
/assign DrAuYueng
Hmmm, in theory yes, since .Next() and .Prev() can return an empty netip.Addr. In practice, they probably never overflow in this particular case. Adding a check for .IsValid() for both Prev and Next would not hurt. I don't expect the path of !ip.IsValid() to be ever taken in these cases, but if they do then that is a bug and it should propagate an error.
/triage accepted


Question: <ins> Shell2Junit License Information Feb, 2010 shell2junit library and sample code is licensed under Apache License, v.2.0. (c) 2009 Manolo Carrasco (Manuel Carrasco Moñino) ===== Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: a. You must give any other recipients of the Work or Derivative Works a copy of this License; and b. You must cause any modified files to carry prominent notices stating that You changed the files; and c. You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and d. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. </ins>

we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


Question: func topologyNormalizingWeight(size int) float64 { return math.Log(float64(size + 2)) } <ins> // adjustForMaxSkew adjusts the number of matching pods in a topology domain // using the constraint's maxSkew. // Topology domains with less than maxSkew number of pods are considered to have // the same priority. func adjustForMaxSkew(cnt, maxSkew int64) int64 { if cnt < maxSkew { return maxSkew - 1 } return cnt } </ins>

Spin-off from What would you like to be : Use maxSkew in TopologySpreadingContraint to affect relative weight of the constraint. Exact formula to be discussed. Some ideas: , intuitive but too coarse-grained at low numbers. Goes: 100%, 50%, 33%, 25%, 20% some linearity. For , goes: 100%, 66%, 50%, 40%, 33% . For , goes: 100%, 75%, 60%, 50%, 42% . Goes: 100%, 70%, 57%, 50%, 44% Why is this needed: When using for , it will be useful for operators to control how strong the constraints should be compared to explicit user requests. Pods would be able to control the weight of constraints per-topology.
/sig scheduling
1 maybe the most compatible option with the current zone:node ratio (2:1):
Not quite. The ratio is correct, but the weight is different. original: prop 1:
I realized that using as weight is quite counterintuitive. Instead, I propose that we use to water down the counts per topology. This way, the behavior still doesn't change for . But as increases, counts become more similar. For example, given the counts and , we get the new counts: . Then we proceed with the counts normalization as usual. Note that when the minimum count is zero in a topology, this procedure has the effect of giving the max score (100) for nodes (or zones) that would satisfy the condition. That won't be true when the minimum count is not zero, but enforcing it would be quite costly. This proposal still gets us closer to that and reduce score differentiation among nodes within the skew. When using constraints against multiple topologies, another side effect is that topologies with bigger would have a reduced effect in the overall score.
That sounds better than the initial proposal which seemed to deviate from the original semantics of the API.
WDYT ?
Sounds good to me. Per constraint on large-scope topology (e.g., zone), I assume it will still be credited with the matching number on its sub topologies (e.g., node), right? (It's just we're using the newCount formula - by taking the maxSkew specified on its constraint into consideration.)

Note that this doesn't replace increasing the weight as described in the parent issue. I think the issue title needs updating.
Understood. Default weight is tracked in another issue. This one talks about Scoring.
Right, just wanted to explicitly mention that to clarify the overall plan since this issue started in the context of increasing the default weight :)
Could you rename that one and clarify that it depends on this issue being resolved first?
can you clarify why that issue is dependent on this one? aren't we going to increase the weight regardless?
I was referring to . But you are right, it doesn't depend on this. kubernetes/enhancements does


Question: } // sanityCheckService performs sanity checks on the given service; in particular, ensuring <del> // that creating/updating a service allocates IPs, ports, etc, as needed. </del> <ins> // that creating/updating a service allocates IPs, ports, etc, as needed. It does not // check for ingress assignment as that happens asynchronously after the Service is created. </ins> func (j *TestJig) sanityCheckService(svc *v1.Service, svcType v1.ServiceType) (*v1.Service, error) { if svcType == "" { svcType = v1.ServiceTypeClusterIP

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should handle load balancer cleanup finalizer for service [sig-network] Services should only allow access from service loadbalancer source ranges Since when has it been failing: 2019-10-19 Testgrid link: https://k8s- Reason for failure: and Anything else we need to know: /sig scalability /sig network could you triage?
It seems that new test was at that point: [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout My personal guess is that we started doing too much load-balancer-related stuff at the same time.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think this is my fault. should fix it. Is there some way to those tests to run against a PR? /remove-triage unresolved
At the moment the fastest way would be to merge your fix and wait for the next run tomorrow.
Reopening just to keep this tracking - will close once we have green run.
the fix helped for a two tests, but one of them is still failing: https://k8s-
I disagree - there are couple loadbalancing-related tests that are flaking with high flakiness ration - I think this PR didn't change anything. So my feeling is that is may still be something like:
Still, the conclusion is that unfortunately more work in needed to put us back in a good shape.
I looked a bit into it and the conclusions are: "Services should handle load balancer cleanup finalizer for service [Slow]" test is consistently failing in small clusters too with: IIUC, the reason is that if we downgrade load-balancer service to non-loadbalaced one, it takes some time to remove the ingress from Status, but it is failing immediately due to this: - can you please take a look? the apps test (and I think couple more) are failing due to incorrect timeouts - I will send out a fix for it soon
Actually, this one will be still failing as of now - the reason for it is described here:

Which jobs are failing: Which test(s) are failing: Since when has it been failing: 18 Oct 13:17 PDT Testgrid link: https://k8s- Reason for failure: check
/sig network /cc /cc
/priority critical-urgent /milestone v1.17
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I believe this is caused by this merge cc.
Most of this is caused by and should be fixed by . Is there any way to run these tests against that PR? /remove-triage unresolved
The job is still failing. Can you check?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage unresolved
OK, fixed some of the tests, got the rest to the point where they failed differently...
Thanks for the update. In testgrid we still see the same error. We will keep observing the next runs.
- still failing. This time the error is We probably need to clean up sanityChecks a bit more - this seems to be a similar issue.
yeah, I'll temporarily un- the tests in my next PR so I can make sure they're actually fixed this time
After the last fix, it seemed to finally recover.


Question: ALL_OS_ARCH = $(foreach os, $(ALL_OS), ${ALL_OS_ARCH.${os}}) CFLAGS = -Os -Wall -Werror -static -DVERSION=v$(TAG)-$(REV) <del> KUBE_CROSS_IMAGE.linux ?= k8s.gcr.io/build-image/kube-cross KUBE_CROSS_VERSION.linux ?= $(shell cat ../build-image/cross/VERSION) KUBE_CROSS_IMAGE.windows ?= dockcross/windows-static-x64 KUBE_CROSS_VERSION.windows ?= latest KUBE_CROSS_IMAGE := ${KUBE_CROSS_IMAGE.${OS}} KUBE_CROSS_VERSION := ${KUBE_CROSS_VERSION.${OS}} </del> <ins> KUBE_CROSS_IMAGE ?= k8s.gcr.io/build-image/kube-cross KUBE_CROSS_VERSION ?= $(shell cat ../build-image/cross/VERSION) </ins> # NOTE(claudiub): The Windows pause image also requires the wincat binary we're compiling for the # port-forwarding scenarios. If it's no longer necessary, it can be removed.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When including windows in a pause build I get a rate limit error as it tries to pull dockcross/windows-static-x64:latest. This causes the build to fail. This image should be stored and pulled from as there should be a copy of this in to mimic kube-cross for Linux images cd build/pause make all-push Kubernetes version : v1.20.1 Cloud provider or hardware configuration: N/A OS (e.g: ): Build running on Linux (includes cross compiling for windows) Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Are you either of you able help with getting this image into the k8s gcr or if there is a reason why it can't be put there??
context - this might be due to use pulling from a small # of vmware VIPs
goal from sig-windows ~ move this into GCR eventually ~ would that be ok ?
do you know any history for this image?
That image carried over from the original work from . I have asked sig-testing previously if we could include that image's functionality into ` into the image. I can try this out in the morning, and if it works, I can send some PRs.
Seems to work. I have sent 2 PRs for this:


Question: return maybeWrapSortingPrinter(cmd, printer), generic, nil } <ins> // PrintResourceInfoForCommand receives a *cobra.Command and a *resource.Info and // attempts to print an info object based on the specified output format. If the // object passed is non-generic, it attempts to print the object using a HumanReadablePrinter. // Requires that printer flags have been added to cmd (see AddPrinterFlags). func PrintResourceInfoForCommand(cmd *cobra.Command, info *resource.Info, f Factory, out io.Writer) error { printer, generic, err := PrinterForCommand(cmd) if err != nil { return err } if !generic || printer == nil { printer, err = f.PrinterForMapping(cmd, nil, false) if err != nil { return err } } return printer.PrintObj(info.Object, out) } </ins> func maybeWrapSortingPrinter(cmd *cobra.Command, printer kubectl.ResourcePrinter) kubectl.ResourcePrinter { sorting, err := cmd.Flags().GetString("sort-by") if err != nil {

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: return true, nil } else if os.IsNotExist(err) { return false, nil <ins> } else if isCorruptedMnt(err) { return true, err </ins> } else { return false, err } } <ins> // isCorruptedMnt return true if err is about corrupted mount point func isCorruptedMnt(err error) bool { if err == nil { return false } var underlyingError error switch pe := err.(type) { case nil: return false case *os.PathError: underlyingError = pe.Err case *os.LinkError: underlyingError = pe.Err case *os.SyscallError: underlyingError = pe.Err } return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE } </ins> // GetSecretForPod locates secret by name in the pod's namespace and returns secret map func GetSecretForPod(pod *v1.Pod, secretName string, kubeClient clientset.Interface) (map[string]string, error) { secret := make(map[string]string)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.

What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: make -C "${KUBE_ROOT}" test  WHAT="${WHAT:-$(kube::test::find_integration_test_dirs | paste -sd' ' -)}"  GOFLAGS="${GOFLAGS:-}"  <del> KUBE_TEST_ARGS="--alsologtostderr=true ${KUBE_TEST_ARGS:-} ${SHORT:--short=true} --vmodule=${KUBE_TEST_VMODULE}"  </del> <ins> KUBE_TEST_ARGS="--alsologtostderr=true ${SHORT:--short=true} --vmodule=${KUBE_TEST_VMODULE} ${KUBE_TEST_ARGS:-}"  </ins> KUBE_TIMEOUT="${KUBE_TIMEOUT}"  KUBE_RACE=""

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I wanted to run an integration test and use the option of to pass parameters to the program being tested. I could not find a way to do so. I tried but this failed because those args were followed by in the actual program invocation (see for the reason). I expected there to be a way. I expected it to be documented. Kubernetes version (use ): current and many recent releases Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/priority important-soon /triage accepted
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale just needs a rebase and another poke at assigned reviewers
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: "fmt" "io" "net" <del> "strings" </del> "sync" "time"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet dies with a panic exception if CSI plugin returns NodeId == "" and nil error. Exception log see at the end of the message What you expected to happen: Kubelet should just raise an error to the plugin and ignore the registration. How to reproduce it (as minimally and precisely as possible): Create a CSI plugin with only Node that would implement GetNodeInfo as follows: Register it as a DaemonSet and see kublet crush Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.10-gke.17 Cloud provider or hardware configuration: GKE OS (e.g: ): Container-Optimized OS (cos) Kernel (e.g. ): - Install tools: Network plugin and version (if this is a network-related bug): Others: Related exception Related line of code: - no check whether error is not nil before calling Error() on it.
/sig storage
/sig node is this the version of the kubelet too? 14 is out of support. is the issue reproducible with a 1.15+ kubelet?
/assign
I checked the code - same defect still exists in current codebase.
thanks /priority important-soon
Wow, that was fast. Thanks!


Question: // the considered volume spec spec     *volume.Spec readOnly bool <del> volume.MetricsNil </del> } var _ volume.Mounter = &flexVolumeMounter{}

kind feature What happened: Currently there is no mechanism to report metrics as the flex plugins are not enabled to collect metrics. All flex volumes lack this feature as compared to in-tree/built-in drivers even though it is possible to collect such metrics for most of the plugins. What you expected to happen: Depending on the availability of the metrics from the mounted FS, flex plugins should allow to report metrics for the FS. How to reproduce it (as minimally and precisely as possible): The default behavior that can be proved easily by just running a command similar to the following command on a node that is running a flex volume. This will show that there are no metrics collected for flex volumes. kubectl proxy $curl 127.0.0.1:8001/api/v1/nodes/<IP/proxy/stats/summary Environment**: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: local x86 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux KubeStorage 4.4.0-130-generic -Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Others: /sig storage
+1 Yes, it will be good, if flex volume can support atleast the default metrics( i.e ), which most of the volume plugins support now. Based upon flex plugin "capabilities" parameters, it can be decided, whether the plugin supports "default metrics" or "nil metrics".


Question: podCount            = 100 podCreationInterval = 100 * time.Millisecond recoverTimeout      = 5 * time.Minute <del> startTimeout        = 5 * time.Minute </del> <ins> startTimeout        = 3 * time.Minute </ins> // restartCount is chosen so even with minPods we exhaust the default // allocation of a /24. minPods      = 50

node-kubelet-serial Kubelet should correctly account for terminated pods after restart Since test was created, flakes 10-20% of the time. Not entirely clear why, I haven't looked. No response /assign /sig node
Introduced by , maybe we should use to replace See the log below, a pod becomes Succeed may exhaust tens of seconds
/triage accepted
Yeah I only tested this locally and not in CI, it seems that in CI the pods take much longer. Timings just need to be tweaked.
/reopen still a number of flakes since the attached PR merged, I don't think that fixed it.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from this link, I see all suites passed, which flakes do you mean?
I see this failure in
so this problem should have been fixed by
From gubernator: In all cases, it's timing out. I'll try upping the startTimeout.
/reopen still flaking
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Luckily the kubelet log is clear as to what's going on. From (flake ) The test should work fine if we drop the CPU request a bit, let me tweak it to make up for the possible extra 100m scheduled. I'm not sure what's causing that (aren't these tests serial? what the heck is getting scheduled) but we should be resilient to it. Also, needs to be fixed. This test will no longer detect the regression it was designed to show after that change. I will revert it and also fix the resource requirements and that should fix this test.
/milestone v1.23 need to fix this before test freeze


Question: } } } <ins> func TestSortInts64(t *testing.T) { src := []int64{10, 1, 2, 3, 4, 5, 6} expected := []int64{1, 2, 3, 4, 5, 6, 10} SortInts64(src) if !reflect.DeepEqual(src, expected) { t.Errorf("func Ints64 didnt sort correctly, %v !- %v", src, expected) } } </ins>

Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks


Question: err = cs.Pods(ns).Evict(e) Expect(err).Should(MatchError("Cannot evict pod as it would violate the pod's disruption budget.")) } else { <ins> // Only wait for running pods in the "allow" case // because one of shouldDeny cases relies on the // replicaSet not fitting on the cluster. waitForPodsOrDie(cs, ns, c.podCount+int(c.replicaSetSize)) </ins> // Since disruptionAllowed starts out false, if an eviction is ever allowed, // that means the controller is working. err = wait.PollImmediate(framework.Poll, timeout, func() (bool, error) {

Kubernetes version (use ): Found in CI Environment: Cloud provider or hardware configuration: AWS defaults What happened: https://k8s- is red, mostly because of the suite. It's probably a timing difference on AWS, but I'm trying to get AWS stable enough for the SQ, and this just isn't. c.f. , , , What you expected to happen: The suite should be green. How to reproduce it (as minimally and precisely as possible): It's constantly being reproduced.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This test is about to enter the blocking category on SQ. Do you have time to look at this, or do I need to blacklist on AWS and find someone to fix the testing hole? I know I don't have the time right now.
how much do we care about this feature working on AWS? Can we disable it for now and fix at our leisure?
Is there anything special about AWS vs, say, on-prem, etc? My guess is that there's something minorly wrong with the timing here, based on looking at and from . It looks like it takes upwards of 20s-30s just to pull the image, so maybe a simple would fix this?
I can't imagine anything specific to the test that would vary based on provider. As you suggest, it's probably something ancillary like that the pods have not started.
Hmm... we actually already wait up to 10 minutes for at least one pod to be running. Perhaps we should wait until all pods are running before proceeding.
Oh, I see now. Yeah, I was misreading the failure, that's just roundtripping the itself.
No. Line 74 is waiting for the PDB to be updated, and it waits only 1 minute. The PDB won't be updated until all the pods are running. I have a PR that waits for all 3 pods to be running before proceeding with this test. Will send shortly.
[FLAKE-PING] This flaky-test issue would love to have more attention.
seems to have driven down the flakes dramatically, but there's a consistent flake on AWS for these tests: Any thoughts?


Question: return fmt.Errorf("error parse ip address: %s", address) } if err := h.AddrDel(dev, &netlink.Addr{IPNet: netlink.NewIPNet(addr)}); err != nil { <del> return fmt.Errorf("error unbind address: %s from interface: %s, err: %v", address, devName, err) </del> <ins> if err != unix.ENXIO { return fmt.Errorf("error unbind address: %s from interface: %s, err: %v", address, devName, err) } </ins> } return nil }

https://k8s- https://k8s- cc: /priority failing-test
Sorry, but I will fix it ASAP.
/assign
PR is already sent. Wating for CI report...


Question: continue } delete(f.items, id) <ins> // Only log traces if the queue depth is greater than 10 and it takes more than // 100 milliseconds to process one item from the queue. // Queue depth never goes high because processing an item is locking the queue, // and new items can't be added until processing finish. // https://github.com/kubernetes/kubernetes/issues/103789 if depth > 10 { trace := utiltrace.New("DeltaFIFO Pop Process", utiltrace.Field{Key: "ID", Value: id}, utiltrace.Field{Key: "Depth", Value: depth}, utiltrace.Field{Key: "Reason", Value: "slow event handlers blocking the queue"}) defer trace.LogIfLong(100 * time.Millisecond) } </ins> err := process(item) if e, ok := err.(ErrRequeue); ok { f.addIfNotPresent(id, item)

I am using client-go/tools/cache package NewInformer, find the StreamWatcher and seeing extremely high memory allocations. Attached is a memory profile: From this we have 2.3 GB of memory allocated by the StreamWatcher. To get to this state I had a script to create pod and delete pod over and over again, and every 10 seconds I run it again. I left this running 30 minutes and saw this memory usage. watch code as follows, cache.NewInformer return controller and storewriter is chan queue, receiving event and distribute to queue
/sig api-machinery
Thanks for taking a look
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
i find that infomer handler be stuck will lead to streamWatcher memory high allocationsdon't hang informer handler and you will resolve that question.
/assign /triage accepted
can you check with a cpu profile if some handler is blocking?
cpu profile is normalas followscode callback handler func (w *Watcher) AddEvent(obj interface{}) { data := w.genSyncData(obj, action.SyncDataActionAdd) if data == nil { return } w.writer.Sync(data) } writer is a buffer chan with length and write chan timeout for one secondprobably is will be stuck when produce lots of eventsfinallyI store events to fifo queue when it produce lots of eventsand consume fifo queue event to writer chanthe issue is solved
So, the problem is that your handlers are slower processing the objects from the delta fifo queue, than the rate at which the events are obtained from the apiserver, this is because of this objects are processed by the handlers locking the deltaFifo queue, hence the streamwatcher accumulates the object in memory until the handlers process the object and Pop() release the lock. The code comments inform about this problem // Add() and Get(), won't be blocked for too long. the only solution I see is logging some errors warning about this (processing withouth locking doesn't seem an option, right?)

I've tried to use queue depth, but that is not an option, since the contention happens before events get into the queue and it nevers goes high. I've tried adding trace for slow handlers operations, however, those maybe harmless if the incoming rate is still lower than the draining rate, unless we decide that we want to trace process handlers that take more than X seconds Is adding another intermediate queue between the streamwatecher and the deltaFifo an option? that will free the streamwatchers ...
Yeah. I don't see it as a high priority, but that could be helpful. Following the standard hander to workqueue pattern, it isn't a practical problem. /priority backlog
i don't hang informer hander by fifo queue, i find that memory will come down when lots of create and delete podsbut i running code onlinethe streamwatcher still exist high memory usage as follows pprof Please take a look at it
i am also facing same issue any guidance for the watcher memory increase in high pod churn


Question: }, }, { <del> name: "kernelVersion is 5.15.0, return safeSysctls with no kernelVersion limit and net.ipv4.ip_local_reserved_ports and net.ipv4.tcp_keepalive_time", </del> <ins> name: "kernelVersion is 5.15.0, return safeSysctls with no kernelVersion limit and kernelVersion below 5.15.0", </ins> getVersion: func() (*version.Version, error) { kernelVersionStr := "5.15.0-75-generic" return version.ParseGeneric(kernelVersionStr)

Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted


Question: <ins> - name: evictions_total subsystem: node_collector help: Number of Node evictions that happened since current instance of NodeController started. type: Counter stabilityLevel: STABLE labels: - zone </ins> - name: framework_extension_point_duration_seconds subsystem: scheduler help: Latency for running all plugins of a specific extension point.

I would like to propose the graduation of the metric from alpha to stable. The metric has been stable for 2 years now.
/sig api-machinery /sig instrumentation
I am willing to do all StabilityLevels from alpha to stable? Is that right?
/assign
- The request is to have the stability level for the metric graduate to stable. If there are other metrics you want to do at the same time that's a bonus, but not part of the request in this issue.
/sig node /remove-sig api-machinery /cc
To be promoted to stable, the metrics would need to be renamed from to in order to follow Prometheus best practices as well as sig-instrumentation naming guidelines.
/cc
/assign
/triage accepted


Question: TestEvent    = framework.ClusterEvent{Resource: "test"} NodeAllEvent = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.All} EmptyEvent   = framework.ClusterEvent{} <del> ) var lowPriority, midPriority, highPriority = int32(0), int32(100), int32(1000) var mediumPriority = (lowPriority + highPriority) / 2 var highPriorityPodInfo, highPriNominatedPodInfo, medPriorityPodInfo, unschedulablePodInfo = framework.NewPodInfo(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "hpp", Namespace: "ns1", UID:       "hppns1", }, Spec: v1.PodSpec{ Priority: &highPriority, }, }), framework.NewPodInfo(&v1.Pod{ </del> <ins> lowPriority, midPriority, highPriority = int32(0), int32(100), int32(1000) mediumPriority                         = (lowPriority + highPriority) / 2 highPriorityPodInfo = framework.NewPodInfo(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "hpp", Namespace: "ns1", UID:       "hppns1", }, Spec: v1.PodSpec{ Priority: &highPriority, }, }) highPriNominatedPodInfo = framework.NewPodInfo(&v1.Pod{ </ins> ObjectMeta: metav1.ObjectMeta{ Name:      "hpp", Namespace: "ns1",

The scheduler cache entered a corrupted state where a Pod, according to the cache dump, was nominated to node A, while the status said it was nominated to node B. This lead to unschedulable low priority pods due to the phantom pod in the nominator cache. The cache to match the Pod status Setup: two nodes with all the resources used. A high priority pod comes in that is able to preempt at least one pod. This might cause these sequence of events X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there. the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Nothing clears this nomination afterwards Usually step 2 happens after 3, so everything works as expected. But in a busy cluster, 3 might be delayed. No response <detailsGKE </details<details</details<details</details<details</details<details</details
/sig scheduling cc
This Pod actually got scheduled: The cache dump was obtained much later at : I think the problem is that we got an update for the Pod (probably the nominated node itself), scheduled it again and assigned a second nominated node for it. The second scheduling attempt failed, but we didn't clear the cache.
/triage accepted
/priority important-soon
Sounds like is related?
It could be the same, or it could be another bug. Note that this is a second scheduling attempt for a Pod that was already scheduled.
My theory of events: X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there.

the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Not sure why this pod update fails, but nevertheless, we have a corrupted cache at this point.
thanks for raising up this and the fix. Could you update the "How can we reproduce it (as minimally and precisely as possible)?" section a bit? Per our offline conversation, the issue will leave a "phantom" pod holding a number of resources in the internal nominator cache. And due to that unnecessary resource reservation, low-priority pods may not get scheduled while they should have been. This might be the key trigger of this issue. In practice, it can be caused by a (way-too-quick) movePodBackToActiveQ call (due to relevant/irrelevant events, or backoff's completion) after step 1 and before step 3.
Updated the description. I will update the PRs on Monday.


Question: if err != nil { t.Errorf("Failed to create PersistentVolume %d: %v", i, err) } <ins> waitForPersistentVolumePhase(testClient, pvs[i].Name, watchPV, api.VolumeAvailable) </ins> } t.Log("volumes created")

That does not make any sense. should not exist at this point, it's from different integration test and starts with . Is it possible that that another test is running at this time? Each test basically starts with , so parallelism is IMO not possible as the tests will delete keys of each other.
So, it looks like the tests are not running in parallel, however some volumes leak from one test to another one, even past . I have no explanation for this, but I can reproduce it locally with logs I need. I have gut feeling that there is something wrong in the way how controller stops - each test starts a new one and stops it in the end. just closes some channels, there is no way how to wait until and finishes. I'll look at it on Monday, any ideas would be welcome.
The tests might be running in parallel - I think that's what does by default, though I'm not certain.
though that parallelism may only be across separate test suites, so is probably not run in parallel.
I bumped the priority to P0 since it is the main cause for kubernetes-test-go now, which blocks merge queue.
This issue is dropping our merge throughput by 60%...
I've made a quick patch which will at least test theory. I'll post it shortly.
Hm, now I see a failure in integration.TestUnschedulableNodes-- will keep my eye on it
There is an open issue for UnschedulableNodes - FYI
thanks for the PR. Did the test fail again after was merged?
I still see some failures of on my machine, fixed by


Question: "fmt" "net/http" "net/http/httptest" <del> "sync" </del> "testing" "time"

We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation. We have the following goals: Goal 1: Save time on testing. More accurate scheduling latency. We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components. Goal 2: Profiling runtime metrics and find out performance bottleneck. Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics. Goal 3: Reproduce test result easily. We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. We have the following plans for writing scheduler tests: density test (by adding a new Go test) schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes print out scheduling rate every second let you learn the rate changes vs number of scheduled pods benchmark (by modifying the existing benchmark in integration or simply add a new one) make use of and report nanosecond/op. schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10. We are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? Any suggestions are welcomed!
/cc
This extends
So density already does this... the one option it looks like you may want is custom control , real or fake is irrelevant as the e2e's just need to point to an api-server. re: benchmarks, seems like we could simply add to the unit-tests.
After chatting with lets add this as an agenda item for the SIG meeting tomorrow.

+1 for adding benchmark (howeveer this should be integration test, not unit-test in my opinion)
cc The next SIG on scheduler meeting is on 12.07, isn't it? If there is one tomorrow, can I have the information please?
retract, +1 to . I was adding this to the list given the context, but it crosses over into both.
Slides from today's meeting:
Here are are the action item notes from the SIG meeting: Everyone agrees this is a good idea and long overdue, but the question is where would this live: New category of tests? (isolated/sandbox’d/component) The current testing structure allows to easily append. create new directory (component) + hack/script to execute Future "we could" gate on known good data, and fail performance regressions
/cc
/cc


Question: if !fp.needFullSync { t.Errorf("Proxier did not fail on previous partial resync?") } <del> updatedPRFailures, err := testutil.GetCounterMetricValue(metrics.IptablesPartialRestoreFailuresTotal) </del> <ins> updatedPartialRestoreFailures, err := testutil.GetCounterMetricValue(metrics.IptablesPartialRestoreFailuresTotal) </ins> if err != nil { t.Errorf("Could not get partial restore failures metric: %v", err) } <del> if updatedPRFailures != prFailures+1.0 { t.Errorf("Partial restore failures metric was not incremented after failed partial resync (expected %.02f, got %.02f)", prFailures+1.0, updatedPRFailures) </del> <ins> if updatedPartialRestoreFailures != partialRestoreFailures+1.0 { t.Errorf("Partial restore failures metric was not incremented after failed partial resync (expected %.02f, got %.02f)", partialRestoreFailures+1.0, updatedPartialRestoreFailures) </ins> } // On retry we should do a full resync, which should succeed (and delete svc4)

When unit tests in are invoked with 1, it fails Unit tests should work for any . This is important for troubleshooting flaking tests. Discovered when I tried to add parallel execution as described in master N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /area kube-proxy
/cc


Question: stream := &fakeUpgradeStream{} c.streams[headers.Get(api.PortHeader)] = stream <del> stream.data = c.portData[headers.Get(api.PortHeader)] </del> <ins> // only simulate data on the data stream for now, not the error stream if headers.Get(api.StreamType) == api.StreamTypeData { stream.data = c.portData[headers.Get(api.PortHeader)] } </ins> return stream, nil }

I0407 16:48:52. ] Forwarding from 6000 -6000 --- FAIL: TestForwardPorts (0.02s) 2: Error reading data: read tcp 127.0.0.1:5000: connection reset by peer FAIL coverage: 57.4% of statements FAIL 0.168s https://travis-
cc
Andy, can you take a look at this one?
I've been unable to reproduce locally (Fedora 21 VM) but I will look again. Dawn Chen wrote:


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package filters import ( "context" "net/http" "k8s.io/apiserver/pkg/endpoints/request" ) type taskKeyType int const taskKey taskKeyType = iota type Task struct { Func func() } func WithTask(parent context.Context, t *Task) context.Context { return request.WithValue(parent, taskKey, t) } func TaskFrom(ctx context.Context) *Task { t, _ := ctx.Value(taskKey).(*Task) return t } // WithRoutine returns an http.Handler that executes preparation of long running requests (i.e. watches) // in a separate Goroutine and then serves the long running request in the main Goroutine. Doing so allows // freeing stack memory used in preparation Goroutine for better memory efficiency. func WithRoutine(handler http.Handler, longRunning request.LongRunningRequestCheck) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { ctx := req.Context() requestInfo, _ := request.RequestInfoFrom(ctx) if !longRunning(req, requestInfo) { handler.ServeHTTP(w, req) return } req = req.WithContext(WithTask(ctx, &Task{})) panicCh := make(chan any, 1) go func() { defer func() { if r := recover(); r != nil { panicCh <- r } close(panicCh) }() handler.ServeHTTP(w, req) }() if p, ok := <-panicCh; ok { panic(p) } ctx = req.Context() if t := TaskFrom(ctx); t != nil && t.Func != nil { t.Func() } }) } </ins>

Optimize the memory usage of goroutine stack for serving watch requests. According to the of handling a watch request, the stack memory is 0x3768, which is very close to 16 KiB and Golang runtime will double it to 32 KiB once exceeded. In fact, it does exceed if we print the traceback deeper and take the fact that extra stack memory reserved by Golang itself into consideration. A naive optimization is to run as much as preparation handlers in a separate goroutine and then move the actual serving to another goroutine, so the former routine can free the stack memory earlier and make it more memory efficient. Given the extensive use of watch requests, 32 KiB per watch could add up to a very high amount of memory overhead for . A typical watch request commonly lasts for 30s or longer, which means this memory will be active for quite long. I see a great improvement in memory efficiency to do this optimization.
/cc
/sig api-machinery
/cc if you would like to take a look when you have time :) Thank you /triage accepted


Question: and minor versions MUST only be an integer. release: v1.19 file: test/e2e/apimachinery/server_version.go <ins> - testname: ControllerRevision, resource lifecycle codename: '[sig-apps] ControllerRevision [Serial] should manage the lifecycle of a ControllerRevision [Conformance]' description: Creating a DaemonSet MUST succeed. Listing all ControllerRevisions with a label selector MUST find only one. After patching the ControllerRevision with a new label, the label MUST be found. Creating a new ControllerRevision for the DaemonSet MUST succeed. Listing the ControllerRevisions by label selector MUST find only two. Deleting a ControllerRevision MUST succeed. Listing the ControllerRevisions by label selector MUST find only one. After updating the ControllerRevision with a new label, the label MUST be found. Patching the DaemonSet MUST succeed. Listing the ControllerRevisions by label selector MUST find only two. Deleting a collection of ControllerRevision via a label selector MUST succeed. Listing the ControllerRevisions by label selector MUST find only one. The current ControllerRevision revision MUST be 3. release: v1.25 file: test/e2e/apps/controller_revision.go </ins> - testname: CronJob Suspend codename: '[sig-apps] CronJob should not schedule jobs when suspended [Slow] [Conformance]' description: CronJob MUST support suspension, which suppresses creation of new jobs.

[X] Test approval issue: [X] Test PR: 8 Jul 2022 [x] Two weeks soak end date: 22 Jul 2022 [x] Test promotion PR: According to this APIsnoop query, there are still some remaining Controller Revision endpoints which are untested. listAppsV1ControllerRevisionForAllNamespacesreadAppsV1NamespacedControllerRevisionpatchAppsV1NamespacedControllerRevisioncreateAppsV1NamespacedControllerRevisiondeleteAppsV1NamespacedControllerRevisionreplaceAppsV1NamespacedControllerRevisiondeleteAppsV1CollectionNamespacedControllerRevision Using a number of existing e2e test practices a new has been created for 7 ControllerRevision endpoints. The e2e logs for this test are listed below. This query shows the following endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/kubernetes/pkg/volume/util/types" ) <ins> const ( Removed     string = "removed" Terminating string = "terminating" Other       string = "other" ) </ins> func pluginPVOmittingClient(dswp *desiredStateOfWorldPopulator) { fakeClient := &fake.Clientset{} fakeClient.AddReactor("get", "persistentvolumeclaims", func(action core.Action) (bool, runtime.Object, error) {

We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.


Question: func (n NamespacedName) String() string { return n.Namespace + string(Separator) + n.Name } <ins> // MarshalLog emits a struct containing required key/value pair func (n NamespacedName) MarshalLog() interface{} { return struct { Name, Namespace string }{ Name:      n.Name, Namespace: n.Namespace, } } </ins>

should implement . This ensures that it's use as value in a key/value pair emits a struct and not the string. /ref
This depends on logr 1.2.0, which will be in /wg structure-logging /sig instrumentation
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : This depends on logr 1.2.0, which will be in /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on this :) /assign
/wg structured-logging
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /lifecycle frozen


Question: # to readonly. # The configured vars will only contain platforms allowed by the # KUBE_SUPPORTED* vars at the top of this file. <ins> declare -a -g KUBE_SERVER_PLATFORMS declare -a -g KUBE_CLIENT_PLATFORMS </ins> kube::golang::setup_platforms() { if [[ -n "${KUBE_BUILD_PLATFORMS:-}" ]]; then # KUBE_BUILD_PLATFORMS needs to be read into an array before the next

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: building Kubernetes release and it failed due to a variable not being set (). It appears a does not function as expected on (used Ubuntu 16). Additional details will be provided in comment below What you expected to happen: successful build of Kuberentes release or documented minimum supported OS releases (i.e., bash versions) How to reproduce it (as minimally and precisely as possible): using Ubuntu 16 (), run [] using , or (containing or subsequent cherry pick) Anything else we need to know?: It appears the issue is with how and create variables, where I believe does not create the variables with the expected scope (created using scope versus scope). This creates a minimum supported OS version when running from the best I can tell. Environment: Kubernetes version (use ):  Cloud provider or hardware configuration: N/A OS (e.g: ):Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): N/A Others: /sig release
Some of the details I mentioned above, modified the file to output some helpful information the diff of changes I made the results when running on Ubuntu 16 (bash 4.3), the variable appears to be scoped inside the function the results when running on Ubuntu 16 (bash 4.3) without the variable attribute (you can now see the variable is a variable, not locally scoped to the function ) And on Ubuntu 18 (bash 4.4) the variable is scoped with the attribute as you would expect and runs as expected.
I guess fixes this problem. cc
I don't think the is/was the only issue here. It was obviously as I was testing with , so only variables were being used and not scoped outside the function (in bash 4.3).
we are also getting the same error on bash version 4.3 cc
any update on this, are you fixing it??
Same question with , this issue impact OpenLab cloud-provider-openstack E2E testing job.
can you please try this?

it works with your fix , previously jobs were failing --- EDIT --- Hold on, I think I was too fast, it failed in a later stage don't know if is a problem or related to the script changes. Should we have the tar files as an output ?
Sorry I missed this yesterday. I will try as well. Otherwise, my assumption was to revert the use of a function, as that creates the scoping problem we've been seeing with variables in .
i think that would be not
Also both not should work now.
Looks like worked for me. Was able to run a using successfully.
ref: as the pr that appears to have caused this


Question: framework.ExpectNoError(err) metrics := map[string]float64{} <del> for method, samples := range response { if metricNames.Has(method) { if len(samples) == 0 { return nil, fmt.Errorf("metric %s has no samples", method) } lastSample := samples[len(samples)-1] metrics[method] = float64(lastSample.Value) </del> <ins> for _, samples := range response { if len(samples) == 0 { continue </ins> } <del> } // Ensure all metrics were provided for name := range metricNames { if _, found := metrics[name]; !found { return nil, fmt.Errorf("metric %s not found", name) </del> <ins> // Find the *last* sample that has the label we are interested in. for i := len(samples) - 1; i >= 0; i-- { metricNameWithLabels := samples[i].Metric.String() if metricNames.Has(metricNameWithLabels) { metrics[metricNameWithLabels] = float64(samples[i].Value) break } </ins> } }

[It] [sig-storage] CSI Mock selinux on mount metrics SELinuxMount metrics [LinuxOnly][Feature:SELinux][Feature:SELinuxMountReadWriteOncePod][Serial] warning is not bumped on two Pods with the same context on RWO volume [sig-storage] [It] [sig-storage] CSI Mock selinux on mount metrics SELinuxMount metrics [LinuxOnly][Feature:SELinux][Feature:SELinuxMountReadWriteOncePod][Serial] warning is bumped on two Pods with a different context on RWO volume [sig-storage] Sample failure: October 25, 2023 It fails since The PR a label to some SELinux metric and the tests do not count with that. No response /sig storage
/assign /triage accepted


Question: } } <ins> func newPodWithPhase(uid, name string, phase v1.PodPhase) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID:  types.UID(uid), Name: name, }, Status: v1.PodStatus{ Phase: phase, }, } } </ins> // syncPodRecord is a record of a sync pod call type syncPodRecord struct { name       string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Restart: Always pods went into OutOfCpu error state after kubelet restart. Pods should remain running as node had sufficient capacity for them. Completed, RestartNever pod should not revert from Completed to OutOfCpu. Launch kubelet with: Run the following script against the local cluster: <details</detailsObserve pod status: Restart the kubelet: Observe pods: xref /sig node Kubernetes version (use ): v1.23.0-alpha.3.183+ - Cloud provider or hardware configuration: Local machine OS (e.g: ) & Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others:
/triage accepted /priority critical-urgent Reproducible on 1.22
From the reproducer: yields However, in the kubelet logs, it sees 7100m cpu allocated?
Looks like the topology admit handler thinks that the completed pods are using CPU which is why when it gets to the 8th it fails it with OutOfCpu. Must be a bug in there somewhere.
/assign
Unclear if could be related but linking for posterity
Confirmed this only affects 1.22, repro on 1.21 has expected behaviour: /sig scheduling I'll start a bisect.
is
Added some logging With this we see I think that the terminated pods are getting filtered, but for whatever reason they are still getting their resource requests to NodeInfo?
points out that my logging language was a bit backwards and that the completed pods should not have been returned, because they already successfully completed and exited. This code is suspect: The logs show that the completed pods are never filtered, so they are failing one of these checks. I am going to try to write a test case that hits this issue so we can test fixes against it.

The completed pods definitely show up as complete, but for some reason they do not get filtered (note I have changed my patch above for the log lines to make sense now): So I guess isPodTerminationRequested=true on already exited pods, but I would only expect that to be true if the pod was actively in the process of termination and not already exited. cc Log patch: <details</details
I tried changing to but apparently it thinks that's true :( Edit: doesn't work either.
The bug is that the only component in the kubelet that is allowed to authoritatively state a container is terminated (pod worker) takes non-zero time to do so during startup (because we have to call kill container to guarantee it's stopped). This races with the pod config loop and admission, which sends all the containers at once. In order for the race to be eliminated, some equivalent of a sync has to be completed during startup on terminal but potentially non-terminated pod workers BEFORE other adds are processed. That could be running HandlePodCleanups once all sources are synced but before HandlePod* methods are called, but HandlePodAdditions() cannot be invoked until after the pod worker has had a chance to verify all terminal pods are actually terminal. EDIT: Another potential approach is for the pod worker to hold starting pods until SyncKnownPods observes all terminal pods are accounted for at least once, then allowing starts.
Also, we appear to be missing a kubelet restart test that verifies admission with completed pods (which I thought was on our list of "missing e2e tests that would have caught these regressions" already). Such a test would have to: pods that consume resources for admission until they are running some of them with failed and` successful states the kubelet some of the running pods the kubelet none of the terminal pods are restarted, all running pods are restarted, and that only non-terminal pods are accounted to admission


Question: for i, pl := range plugins { weight := f.scorePluginWeight[pl.Name()] <del> nodeScoreList := pluginToNodeScores[i] </del> <ins> nodeScoreList := pluginToNodeScores[pl.Name()] </ins> score := nodeScoreList[index].Score if score > framework.MaxNodeScore || score < framework.MinNodeScore {

/sig scheduling /kind regression caused a performance degradation in the 5k nodes scalability tests ! The PR changed how to split the load among parallel threads. Previously, we would parallelize over the number of nodes. The PR does it over the number of plugins, which is much smaller. Performance to improve or remain the same https://perf- No response 28 N/A <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign The quickest fix is to revert, but I'm open to other suggestions.
/priority critical-urgent
Can we just use another Parallelizer to handle the score process for each plugin? I ran the UT locally and it seems fine, but I'm not familar with
Depending on how many cores you have available, you might just overload the routines. I think it's better to revert the parallizer over plugins while keeping the change from map to slice. Although I don't expect a major impact for the latter. You might want to run a profiler for the benchmark with a higher number of nodes.
I'll create a revert version first :)
Let's revert this first since we reached the code freeze. We can do the refactor later.
Sorry for causing this trouble and thanks everyone for a prompt action


Question: } pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ <del> // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name, </del> <ins> // Must be unique per disk as it is used as the unique part of the // staging path Name: fmt.Sprintf("%s-%s", AWSEBSDriverName, ebsSource.VolumeID), </ins> }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{

Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news.

your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


Question: // Accept calls the Visit method on visitor that corresponds to elem's Kind func (elem GroupKindElement) Accept(visitor KindVisitor) error { <del> if elem.GroupMatch("apps", "extensions") && elem.Kind == "DaemonSet" { </del> <ins> switch { case elem.GroupMatch("apps", "extensions") && elem.Kind == "DaemonSet": </ins> visitor.VisitDaemonSet(elem) <del> return nil } if elem.GroupMatch("apps", "extensions") && elem.Kind == "Deployment" { </del> <ins> case elem.GroupMatch("apps", "extensions") && elem.Kind == "Deployment": </ins> visitor.VisitDeployment(elem) <del> return nil } if elem.GroupMatch("batch") && elem.Kind == "Job" { </del> <ins> case elem.GroupMatch("batch") && elem.Kind == "Job": </ins> visitor.VisitJob(elem) <del> return nil } if elem.GroupMatch("", "core") && elem.Kind == "Pod" { </del> <ins> case elem.GroupMatch("", "core") && elem.Kind == "Pod": </ins> visitor.VisitPod(elem) <del> return nil } if elem.GroupMatch("extensions") && elem.Kind == "ReplicaSet" { </del> <ins> case elem.GroupMatch("extensions") && elem.Kind == "ReplicaSet": </ins> visitor.VisitReplicaSet(elem) <del> return nil } if elem.GroupMatch("", "core") && elem.Kind == "ReplicationController" { </del> <ins> case elem.GroupMatch("", "core") && elem.Kind == "ReplicationController": </ins> visitor.VisitReplicationController(elem) <del> return nil } if elem.GroupMatch("apps") && elem.Kind == "StatefulSet" { </del> <ins> case elem.GroupMatch("apps") && elem.Kind == "StatefulSet": </ins> visitor.VisitStatefulSet(elem) <del> return nil </del> <ins> default: return fmt.Errorf("no visitor method exists for %v", elem) </ins> } <del> return fmt.Errorf("no visitor method exists for %v", elem) </del> <ins> return nil </ins> } // GroupMatch returns true if and only if elem's group matches one

What happened: There are many if cases in kind visitor. What you expected to happen: Use switch instand of more three if cases.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I sent a PR to solve the issue and the PR is wait for merging.


Question: ``` --client-certificate="": path to client-certificate for the user entry in kubeconfig --client-key="": path to client-key for the user entry in kubeconfig <del> --embed-certs=false: embed client cert/key for the user entry in kubeconfig </del> <ins> --embed-certs[=false]: embed client cert/key for the user entry in kubeconfig </ins> --password="": password for the user entry in kubeconfig --token="": token for the user entry in kubeconfig --username="": username for the user entry in kubeconfig

It acts like a boolean flag but requires an argument, unlike boolean flag is pretty much every environment. We need to find a better way to express the things that this is expressing. Maybe a add a tristate to the pflag lib that behaves the same way but doesn't demand an argument?
cc
The handles "true", "false", "unset" and has a default value. I seem to recall you were looking at changing it, but we needed to preserve the value. The complain here is that you have to do instead of just . /cc
fixing is really easy....


Question: if mockReplenish.namespace != oldService.Namespace { t.Errorf("Unexpected namespace %v", mockReplenish.namespace) } <ins> mockReplenish = &testReplenishment{} options = ReplenishmentControllerOptions{ GroupKind:         api.Kind("Service"), ReplenishmentFunc: mockReplenish.Replenish, ResyncPeriod:      controller.NoResyncPeriodFunc, } oldService = &api.Service{ ObjectMeta: api.ObjectMeta{Namespace: "test", Name: "mysvc"}, Spec: api.ServiceSpec{ Type: api.ServiceTypeNodePort, Ports: []api.ServicePort{{ Port:       80, TargetPort: intstr.FromInt(80), }}, }, } newService = &api.Service{ ObjectMeta: api.ObjectMeta{Namespace: "test", Name: "mysvc"}, Spec: api.ServiceSpec{ Type: api.ServiceTypeNodePort, Ports: []api.ServicePort{{ Port:       81, TargetPort: intstr.FromInt(81), }}}, } updateFunc = ServiceReplenishmentUpdateFunc(&options) updateFunc(oldService, newService) if mockReplenish.groupKind == api.Kind("Service") { t.Errorf("Unexpected group kind %v", mockReplenish.groupKind) } if mockReplenish.namespace == oldService.Namespace { t.Errorf("Unexpected namespace %v", mockReplenish.namespace) } </ins> }

master-informing: capz-windows-containerd-master capz-e2e.[It] Conformance Tests conformance-tests ! 2023-03-10 04:56:51 +0000 UTC No response /sig testing
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### Which jobs are failing? ### Since when has it been failing? /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /triage accepted
no major change between passing test on 3/9 17:55 PST and failing test on 3/9 20:56 PST
containerd logs have error about failing to unpack azure-cloud-manager on both nodes:
kubelet logs confirm this:
looks to be related to bumping capz to 1.8 which uses the cloud-provider oot image: caused by upgrading to 1.8 capz in combination with using OOT
This looks to be affecting our release jobs as well:
looks like the image being built is linux only () and is being pulled by windows. When setting up the snapshot in containerd it is failing (as it is missing files required by windows)
For the passing test the image is multi-arch manifest, so the windows image pulls properly (azure-cloud-node-manager:)
There is a missing env in the see


Question: apiserver_dropped_requests_total{request_kind="mutating"} 1 # HELP apiserver_request_total [STABLE] Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code. # TYPE apiserver_request_total counter <del> apiserver_request_total{code="429",component="apiserver",dry_run="All",group="batch",resource="jobs",scope="cluster",subresource="status",verb="PATCH",version="v1"} 1 </del> <ins> apiserver_request_total{code="429",component="apiserver",dry_run="All",group="batch",resource="jobs",scope="resource",subresource="status",verb="PATCH",version="v1"} 1 </ins> `, }, }

I've observed, that some metrics has and labels. (create) requests are "single resource", therefor they should be classified as . Def: customresourcehandler uses , which is based on the RequstInfo only: the RequestInfo is based only on the requests: method and url example: e.g: Since this RequestInfo has Name and Namespace empty, it is classified as . In a similar way it could be explained why: There's a mismatch in scope classification between built-in and custom resources: The key difference - the check if either "name" was provided OR it is a : Where the metrics.CleanScope ONLY checks, if name is not-empty. Tested: = v1.20 There were other scope related issues <1.20 and the distinction is not obvious. <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted