Question: } folder := object.NewFolder(f.client, ref.Reference()) <ins> // Set the InventoryPath of the newly created folder object // The default foler becomes the datacenter's "vm" folder. // The "vm" folder always exists for a datacenter. It cannot be // removed or replaced folder.SetInventoryPath(path.Join(f.dc.InventoryPath, "vm")) </ins> return folder, nil }

Expectation- We should be able to create PVCs via a storageclass with storagePolicyName defined while using a singed ca cert for secure communications with vCenter. Bug- The PVC created with a storageclass with storagePolicyName defined fails with a error when PVCs created with a storageclass without the storagPolicyName work against the same vCenter and datastore. Scenario: Utilizing the vSphere in-tree plugin on kubernetes 1.12.7. Utilizing secure communication with vSphere (signed ca cert imported into the cluster) We are able to create PVCs with the following storage class: However, if we include a storagePolicyName we receive a failure: Here is the yaml for our sc1 storage class: /sig vmware
I think we are having issue with creating PBM client using certificate set on the vCenter Client. We need to create a PBM client to get list of compatible datastore for given storage policy. I think call is failing at - Can you please provide controller-manager logs for further debugging. Make sure log level is set to 4.
Sorry for the delayed response. I'm not confident I got the log level correct, however this is what I find:
I believe I have log level 4 now set:
For comparison, here is a PVC created w/o storagePolicyName on the same datastore which succeeds:
thanks for posting logs. Now this is confirmed, issue is happening while creating PBM client when we use certificate based authentication. One more thing I want to confirm regarding the certificate, are you using solution user as mentioned here -
Can you also provide value of in the file? Is it set to true or false / 1 or 0 ? Also confirm version of vCenter you are using.
is unset, as for the solution user I'd have to leave that to to expand on.
I don't believe we used any solution user because we simply used "" as the default user during installation. please correct me if I'm wrong.
How is the certificate and private key retrieved for the user . I am successfully able to connect to the PBM using vpxd certificte and key. I ssh into the vCenter and executed following commands to retrieve certificate and private key for vpxd. with vpxd cert and private key, I do not see this issue. In addition I also set while making connection.

Can you try setting Insecure to false? That's the flag setting that triggers the cert error.
with , I see the exact same error - , while creating VC client.
Isn't that because it's self-signed? In the case of the issue in this ticket the cert is signed by a trusted CA /cc to confirm.
The cert is self-signed and its CA isn't trusted.
I used vpxd certificate and issuer is I require to set .
Is there any way to make it work if insecure=false?
- then I’d say that’s expected? If the root CA isn’t trusted, then it is insecure?
well, while that's expected, I don't know if there's a method to add the root CA to a trusted list.
Correct me if I’m wrong but isn’t the trusted list the root CA store of the K8S node’s OS in this case? I’d imagine whatever CA Root signed the cert would need to be to the OS of choice’s CA store.
you are correct, we need to download trusted root certificate and add them in the store on node vm. For now on my mac, I have downloaded trusted root certificate from the vCenter -and them in the keychain and flagged as "Always Trust". ! After this, I am able to connect to VC client with or without specifying this flag at all using VPXD certificate and private key. So certificate needs to be to the CA trust store on the master node VM. after adding trusted root certificate from vCenter to the trust store, you should be able to make a connection using .
After importing the root cert by placing it in /etc/ssl/certs on the master node, I'm able to provision a PVC while specifying a storage policy. Thanks for all the help.
Can you close this issue and work with KK to get this automated in the product?
It turns out that importing the root cert into the host was just a workaround, and the actual bug was fixed in That fix is included in govmomi v0.21.0.


Question: } type quotaTestCase struct { <ins> name                             string </ins> path                             string poduid                           types.UID bytes                            int64

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: } actions := filterInformerActions(f.client.Actions()) <del> informerActions := 0 </del> for i, action := range actions { <del> if len(f.actions)+informerActions < i+1 { </del> <ins> if len(f.actions) < i+1 { </ins> f.t.Errorf("%d unexpected actions: %+v", len(actions)-len(f.actions), actions[i:]) break } <del> expectedAction := f.actions[i-informerActions] </del> <ins> expectedAction := f.actions[i] </ins> if !expectedAction.Matches(action.GetVerb(), action.GetResource().Resource) { f.t.Errorf("Expectednt%#vngotnt%#v", expectedAction, action) continue

https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment} Previous issues for this test:
I hit it too, direct link to build: https://k8s- PR here:
: can you try rebasing on ? That should have fixed this problem
Looks like they finally passed and retest is skipped for the submit-queue for that PR, so I'd prefer not to rebase and risk encountering another flake. If you need it as a confirmation of , let me know and I'll force push my rebase.
no worries. things are looking up in the gate. we can close this issue out tomorrow
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
This is still happening and blocking the build. please treat this as a real P0.
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
cc
opened


Question: return lbStatus.Ingress[0].IP, nil } <del> func (az *Cloud) ensurePublicIPExists(service *v1.Service, pipName string, domainNameLabel, clusterName string, shouldPIPExisted bool) (*network.PublicIPAddress, error) { </del> <ins> func (az *Cloud) ensurePublicIPExists(service *v1.Service, pipName string, domainNameLabel, clusterName string, shouldPIPExisted, foundDNSLabelAnnotation bool) (*network.PublicIPAddress, error) { </ins> pipResourceGroup := az.getPublicIPAddressResourceGroup(service) pip, existsPip, err := az.getPublicIPAddress(pipResourceGroup, pipName) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: ns := "not-the-default" testSimpleCRUD(t, ns, noxuDefinition, noxuVersionClient) <ins> testFieldSelector(t, ns, noxuDefinition, noxuVersionClient) </ins> } func TestClusterScopedCRUD(t *testing.T) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: got an error when trying to use a fieldSelector when watching a custom resource What you expected to happen: no error How to reproduce it (as minimally and precisely as possible): a CRD and then a CR for it. to watch with a fieldSelector, such as Anything else we need to know?: The customresource implementation has a defined, and it looks like it should support at least a field selector for and , but I'm guessing we need a bit more in place to make this work for . Environment: Kubernetes version (use ): HEAD (commit ) /sig api-machinery I can try to submit a PR to fix once I finish my current work item, if nobody picks this up sooner.
/area third-party-resource
cc
This is a more general form of . I don't have immediate plans to work on either of them, so you're welcome to take it.
sounds good. Do you want me to close this as a dupe?
I think there's a useful distinction between the two issues. If we support only , that would be enough to but this one tracks the idea of supporting arbitrary field selectors. By the way, if I recall correctly, when I applied the hack mentioned in the other thread to skip the step for CRDs, I think I saw that actually arbitrary field selectors worked too (not just ). That would mean enabling this might be easy on a technical level, although we may need to think through whether this could tie our hands if we ever wanted to add versioning support for CRD in the future. This issue is a better home for that discussion than the other one.
This would help a lot with debugging in prow. Today, the only thing we can do is bubble up information from the spec of a prowjob as labels and use a label selector.
What about just having 's just return the passed in label and value without any "conversion" for now?
I don't think we'd want to do it for all Unstructured, since there's no guarantee all Unstructured objects are CRD objects. However, it should be pretty easy to arrange for ConvertFieldLabel to be a no-op for CRD objects, which makes sense currently because CRD only allows one version for a given resource.

We just need to make sure this will not prevent us from doing more with CRD versioning in the future. My suspicion is that it should be fine, but I haven't investigated properly.
ok, I can put together a PR to make this change. any gut reactions?
PR:
I am sorry to comment on this closed thread. Currently, I am working on a CRD and needs to support arbitrary field selectors. I've tried to do it by learning from what you do for . But it always throws error: . So, I am wondering if I am doing something wrong or it only supports limited field selectors for CRD, which is stated . Here is my code snippet:
CRDs still only support namespace and name for field selectors
Any plan to support arbitrary field selector for CRDs?
Not currently

Follow a TPR an instance a single instance return 400. (watch a collection of TPR works) master and kubectl version: 1.7.1 CRD has the same issue. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I was able to reproduce this with CRD on master. The problem is that kubectl implements a watch on a single object as a watch on the collection with a field selector that filters by . However, field selectors don't work on CRD objects because they are stored as , which does not support : This method is used to transform the field selector from versioned to internal field names: As a test, I bypassed that check for Unstructured objects, and watches on single CRD objects started working. Given that CRD doesn't support multiple versions anyway, it seems feasible to support this with a properly-targeted override of specific to CRD objects. cc
If we could support name and namespace generally on all objects containing object meta, I think I'm in favor of the concept.
I think it's feasible to fix this for CRD at some point. However, given that TPR is already gone in master, I don't think it's worth fixing for TPR unless it's somehow a regression in 1.7. Do you have any objections to that?
Having CRD support is good enough for me and service catalog. Will you backport the fix for CRD to 1.7?
At this stage I wouldn't expect that since it isn't a regression (I don't think) for either extension mechanism.
cc
I am also getting the same error when i try to watch CRDs and filter on a specific field. Im a currently using the to watch for CRDs. On a related note, i noticed that the takes a . Is there a way to pass a label selector so that i can watch CRDs using labels?
Also is this issue slated for 1.8?

(For CRDs that is)
The general issue of supporting field selectors for CRD is tracked in . I don't think it's slated for 1.8. The methods on should take a , which can contain an optional .


Question: return cmd } <del> func (o *VersionOptions) Complete(f cmdutil.Factory, cmd *cobra.Command) error { </del> <ins> // Complete completes all the required options func (o *Options) Complete(f cmdutil.Factory, cmd *cobra.Command) error { </ins> var err error o.discoveryClient, err = f.ToDiscoveryClient() // if we had an empty rest.Config, continue and just print out client information.

Anyone itching to understand the code and try fixing things in Kubernetes, please help fixing the lint failures Step 1: Remove one or two packages (related packages) from the list in - (note: do not change generated files, the name of defaulting functions, or the name of conversion functions... packages failing linting due to issues in those files will need to be left alone) Step 2: Run and look at the logged errors Step 3: Fix the code to ensure that runs cleanly. Step 4: Run and to make sure we fix any problems with gofmt or bazel. Step 5: File a PR with the change Step 6: Go to step :) Please use the tracking spreadsheet For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
/sig architecture /sig testing
Hey, I would like to resolve this issue. Could you assign it to me ?
there are a lot of packages so there will be many people working on this. Also we cannot assign issue to folks who are not in the github org for kubernetes. please feel free to pick up a few packages like i mentioned and file a PR. Make sure you link back to this issue, so folks can see which packages you filed PR for.
OK. I see. I will firstly pick up the package called to fix.
Hi dims. Should i submit my PR before i remove one or two packages (related packages) from the list in - Or I first remove the package from list, then I fix the issue and submit a PR?
in your PR, you will have both the fixes to packages and some lines removed from /.golint_failures
When I run , I found some errors as below: I found that there is a go file in But there is still a error.
Say your GOPATH looks like this Your kubernetes directory should be () cd to and then run from that directory.
Is it necessary to obey all rules in golint? Rules in golint are set in stone. Maybe using gometalinter to ignore some specific errors is better?
I revise some erros about golint in some packages. Please review it.
which errors would you like to ignore?

if you can open a new issue with the list of errors to ignore (and proposing gometalinter), we can get some opinions there from others in the community.
Let me take the kubectl cmd part...
Can someone please review and let me know if I am doing it right?
I ll work on all the packages related to
we get 800-ish pkgs left to fix. my concern is that if we fix them package by package, then such pulls will be flooding the repo. can't we possibly make it in larger pulls? the larger the better?
+1 to slightly larger pulls with related packages. if allow too large then the reviewers will not be able to do a thorough review. we should spread the load among multiple reviewers. (if we allow a really large pull, we will end up with top level reviewers who will face the burden and the PR(s) will just stagnate)
oh good point. we could put one pkg's change into one commit, and merge all changes under one area into one pull so that it wont spread the load. wdyt? for each pull, we can have hundred-ish lines of change. but it will still be 30~50 pulls i guess? good news is that we can take our time to deal with it, at least before 1.12
sounds great! please feel free to help make it happen :) (by suggesting groups of packages that kinda go together)
I'm looking at fixing pkg/proxy, and I think I see what was talking about ignoring some rules. proxy.ProxyProvider is being suggested because golint is saying that it stutters. In that example, I think it's okay renaming it to proxy.Provider, but I could see an example where you would want to keep the stutter if the package that it's declared in uses it heavily.
sure thing, feel free to propose a PR for switching to gometalinter independently of other patches. We can work through that there.
I sent the pull request to fix pkg/volume/aws_ebs
I've fixed a few of these, but get the following error when I try to run : Are there any instructions on what I need to setup in order to resolve this?

you can find this file in
I have raised a PR to fix golint errors in pkg/api
I have fixed some pkgs under /pkg/kubelet pkg/kubelet/images pkg/kubelet/kuberuntime pkg/kubelet/leaky pkg/kubelet/pleg
I submitted a pull request for golint errors in pkg/security/ edit: I can work on picking up pkg/controller/bootstrap through pkg/controller/namespace/... (114-129)
fix golint errors in package cmd/kubeadm/app, since the codes under cmd/kubeadm/app/apis/kubeadm/v1alpha2 and cmd/kubeadm/app/apis/kubeadm/v1alpha3 are auto-generated, i keep them untouched.
I am working on some packages under pkg/kubelet, anyone interested in this plz leave messages for me :)
fix golint for pkg/util/dbus pkg/util/labels pkg/util/node pkg/util/strings pkg/util/workqueue/prometheus
PR fixes golint errors in pkg/routes,pkg/securitycontext, pkg/serviceaccount
PR fixes golint in pkg/probe
should we have a checklist as part of the issue summary saying what is in progress and what is done? Since this issue is getting long, this will make new contributors to pick up issues easier.
Great idea!, want to throw a google doc with a check-list so it's easy for participants to organize themselves? (please!) If it is the issue summary then one of us with privileges will be a bottle neck.
Created a with all the package that have on going PRs. Maybe you can add this to the summary. For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
Thanks updated the spread sheet
Raised PR to fix golint issues in: pkg/controller/clusterroleaggregation pkg/controller/cronjob pkg/controller/daemon pkg/controller/deployment pkg/controller/deployment/util pkg/controller/disruption pkg/controller/endpoint
This issue is close in accident by . could you reopen it?
/reopen
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I found a problem trying to fix golint messages for .

Golint complaints on 's function because of the snake case. When I try to fix it, fails because it is being used in a code generated file. If I clean generated files, tests pass, but the generated file is not invoking it anymore. I failed to find the generator definition using this so I could change the hook to just . Also I'm not sure if this is wanted. I also found other packages in also probably affected by this. I read somewhere about replacing golint with gometalinter, which would allow to define directives in code to omit certain rules. I would consider to have a second exclusion file for golint with these cases, to be updated if move to gometalinter is finally done. This would make this issue completion more feasible. Meanwhile, I'll move onto fixing golint messages for other packages, until this discussion gets cleared.
I also found a problem fixing . A test fails because the directory is marked as readonly (see tests @ Shall we also ignore this directory for this issue?
Is this issue still relevant because hack/verify- is giving me no errors.
the failures are still present in there are details in the first message on how to claim + solve issues :)
Sorry I misunderstood the issue
I'm working on this package:
pkg/apis/admission/v1beta1 pkg/apis/authentication/v1beta1 pkg/apis/authorization/v1beta1 pkg/apis/coordination/v1 pkg/apis/coordination/v1beta1 pkg/apis/imagepolicy/v1alpha1 pkg/apis/scheduling/v1alpha1 pkg/apis/scheduling/v1beta1 pkg/apis/settings/v1alpha1 pkg/apis/storage/v1alpha1
pkg/cloudprovider/providers/vsphere/vclib pkg/kubelet/apis/deviceplugin/v1alpha pkg/kubelet/apis/pluginregistration/v1alpha1 pkg/kubelet/apis/pluginregistration/v1beta1
:
I'm looking at the pkg/registry/*, all those which are left. I expect to give an update on what all is fixed in the next 24-48 hours.
hi there is a google sheet that is a bit outdated, maybe you can update that?
hi I seem to only have view access for the sheet. Can't edit.
I fixed the permission.

Please try it again
pkg/apis/authentication pkg/apis/coordination pkg/apis/events pkg/apis/imagepolicy pkg/apis/settings
hi I think this issue was closed prematurely by (using fixes #issue in commit)
I'll put in a PR shortly for the package
Looking at next
hi thanks! there is a spreadsheet here that tracks the progress on fixing these issues, consider adding the packages you're work on there so we can keep everything organized :+1:
Starting on now
I did indeed see the Google Sheet link (thank you and to take it a little further
test/e2e
test/e2e_node/builder
pkg/registry/admissionregistration/mutatingwebhookconfiguration/storage pkg/registry/admissionregistration/rest pkg/registry/admissionregistration/validatingwebhookconfiguration/storage pkg/registry/apps/daemonset/storage
Hi. I opened a PR - Please review.
Working on Will open a PR soon.
Starting on and should have a PR soon
I keep getting this golint verify error on and was wondering if others saw it: But when I that package to the golint failures in my PR, my reviewer said it isn't necessary? What does that mean? What did I miss? Should I use a different kind of verify? cf:
looks like the disconnect is because that problem does not show up in the version of we use in the CI. please see
Still confused: is that not the golint version I ran? it installs from vendor and uses that I think?
i installed from and i still don't see the problem, can you please check what version of golint you have?
I'm confused why I wouldn't be using the vendored version? how would that get out of sync?
/me looking
Note: I am reserving for a KubeCon Contributor Workshop demonstration. It is noted in the spreadsheet.
I think may not be ideal cause there's a lot of auto generated files there. What do you think about reserving instead for the KubeCON Contributor Workshop?
pkg/apis/scheduling pkg/apis/storage/util pkg/apis/storage/v1beta1/util pkg/apis/storage/v1/util
/priority backlog
Hi I am a new contributor here, I'd like to contribute and make changes to the following file/s. I have marked it as WIP in the Google Sheet. Can someone let me know if I have to do anything else before I start working on this PR?

Umm so there's an issue where Golint tells that underscore shouldn't be allowed. Here's a log of the go-lint warnings:
Thanks The following should not be modified, as they are designed like this for code generation.
I'd like to pick a package and do my first PR, is there a list of packages to be done? Or should I correlate with this issue thread? Thanks in advance!
please see step 1 () please scan other PRs linked to this issue as well just in case.
Will do. Thank you. On Thu, Oct 3, 2019, 7:25 AM Davanum Srinivas <:
Hi there, I'm looking for a good first issue. I'll start taking a look at if that's ok? There doesn't seem to be anyone currently tracked as looking at it in the attached spreadsheet.
I've found a few others I'm actively looking at, you won't get any argument from me. On Sat, Oct 12, 2019, 10:38 AM Pamela Kelly <:
I am working on the iptables pkg. Updated in the google docs
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have problem running in step 2 $ hack/verify- what should I do?
PTAL for my pr to this issue, thx
you can move kubernetes directory to and re-run this script.
I'd like to take Can we fix linter issues for generated files? Or we should skip packages with zz_generated?
There're 2 lint errors that affects 292/507 packages, meaning it's not possible to cleanup the entire file without solving it: default pattern of Kubectl is the usage of [cmd]Options for initial options structs and it raises: /apis/{custommetrics,externalmetrics} for example raises the underscore error, but It's probably not worth to break backward compatibility renaming these folder for lint error fix: Any advice on these?

/reopen hm, cannot reopen this issue, it was closed by mistake
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am sure this is all not fixed. Am I missing something? ./hack/verify- installing golint from vendor Congratulations! All Go source files have been linted.
you need to remove package from first.
/remove-sig testing SIG testing doesn't own code failing lints, we own the linter. :upsidedownface:
Hello, is this still a relevent issue ? I would like to work upon
I have fixed
Hi Folks, Some of the golint fixes, require us to change the exposed function's name, in simple words, "change API". I was not sure, if that is welcomed and what is the process.
Heres what Im getting after going over the following steps as mentioned by Say your GOPATH looks like this GOPATH=/Users/dims/go Your kubernetes directory should be ($) cd to and then run hack/verify- from that directory. I get the following output - installing golint Congratulations! All Go source files have been linted. Im not really sure what I am doing wrong. Any help is appreciated!
I think I understand. I misunderstood the steps around .golint_failures. The first step is to remove the pkg that we are targeting from this file and then run the My mistake was to assume that the packages in the file need to be removed after we are done cleaning the golint errors.
Can someone please review and let me know if I've done this right?
After a couple years of seeing changes spawned from this issue, I don't think this is a good use of contributors' or reviewers' time.

golint is very opinionated about: exported type and function names, leading to PRs attempting to change exported things people use downstream missing documentation on interface implementation functions, which leads to not-very-useful copy/paste of interface godoc the govet and staticcheck verify jobs are much more useful since they catch correctness issues, but I would recommend closing this issue and dropping the golint checks from CI golint doc itself discourages from using its output in an automated enforced way: from
i guess one can say gofmt is opinionated too, but something good about Golang is that it comes with an opinion and a set of standard tools. for API documentation at least we found inconsistencies on how most exported API items are documented: golint can help with that, but if there are no "pragmas" to avoid breaking changes and given we have codegen generating non-compliant names enabling golint on API packages is sort of blocked in Kubernetes. for non-API, non-library, non-public code, i think being inconsistent is mostly OK. no preference on my side. shrug
true, but code formatting has the benefits of mostly eliminating meaningless diffs in PRs due to code movement/whitespace/etc if there are specific packages (like API packages) we lack documentation on, adding checks specifically for that could make sense exactly, I don't think golint is the right tool for making sure we have API docs
golint itself doesn't have these and the go project is against this sort of comment AIUI, however golangci-lint does have them. I think there are plenty of other static analysis lints we should use instead, beyond just staticcheck and vet there is e.g.
btw, if we remove golint can we see e.g. all-caps constants or functions with underscores dodging review? perhaps we should add golangci-lint soon after:
Completely agree there are better places to spend review bandwidth. I was curious to see what the effort looked like. There have been 890 commits that touched over the past 3+ years !
/remove-help /remove-good-first-issue
Overall agree on the bandwidth. Also note that it was a good first issue with very low bar to get them in the repo. May be we need to find a similar low bar, but high value good first issue.

UT's come to mind, are there other things we curate for new contributors ?
Do we have an alternative way of enforcing that new exported code (not necessarily API fields) has at least some minimum documentation? As a reviewer, I ask contributors to add meaningful comments. But reality is that I sometimes stumble into old code that doesn't have documentation because we didn't have golint at some point. I agree that fix-golint PRs just add burden to reviewers. But, before removing the lint check, can we wait and see if it's enough to discourage people from sending more PRs like this? Additionally, we can encourage reviewers to just close PRs that only fix golint issues.
+1 for adding burden to reviewers. However, as a new contributor, I think this a very good opportunity for new contributors to study the code base and submit their first PR. In my opinion, I think this issue would be able to attract more new contributors, get familiar with the code base, and get familiar with the git workflow.
thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: At some point we can look at adding golangci-lint, there's a WIP in progress in /close
Closing this issue. <detailsIn response to : thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: panic(err) // TODO: Propagate error up } <del> // We explicitly do NOT do any decoration here - switching on Cacher // for events will lead to too high memory consumption. opts.Decorator = generic.UndecoratedStorage // TODO use watchCacheSize=-1 to signal UndecoratedStorage </del> store := &genericregistry.Store{ NewFunc:       func() runtime.Object { return &api.Event{} }, NewListFunc:   func() runtime.Object { return &api.EventList{} },

Currently we signal undecorated storage by simply affecting the undecoratedStorage to the decorator. This is a blocker for kubernetes/enhancements as we need to enable watch cache for events (which currently hardcoded and not configurable) cc /assign /sig api-machinery


Question: return false } <del> accessModes := c.spec.PersistentVolume.Spec.AccessModes </del> <ins> if c.spec.PersistentVolume == nil { klog.V(4).Info(log("mounter.SetupAt Warning: skipping fsGroup permission change, no access mode available. The volume may only be accessible to root users.")) return false } </ins> if c.spec.PersistentVolume.Spec.AccessModes == nil { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return false } <del> if !hasReadWriteOnce(accessModes) { </del> <ins> if !hasReadWriteOnce(c.spec.PersistentVolume.Spec.AccessModes) { </ins> klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return false }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet panics on creation of inline ephemeral volume with FSType specified. creation of inline ephemeral volume succeeds with FSType specified. no existing isolated test case. Happens during e2e testing of pmem-csi. Kubernetes version (use ): The pmem-csi CI system where this was detected, runs same tests on k8s systems with 1.19 and 1.20. The problem occurs in testing on 1.20 system, problem does not occur on 1.19 system.Cloud provider or hardware configuration: pmem-csi tests run on cluster formed by few qemu-based systems running Fedora-33 and emulated pmem storage. The issue has been seen in e2e testing in , where a new test was recently introduced that creates a pod with an inline ephemeral volume. If the related VolumeSource has filesystem type set to empty "", or FSType is omitted in spec, then the pod creation works OK. In that case, pmem-csi chooses it's default fstype which is . Example of working definition of VolumeSource: If the filesystem type in the VolumeSource is set to or then kubelet fails to create the volume and pod. A panic trace is written in the node kubelet log and the creation will be retried few times, On pmem-csi side: the 1st NodePublishVolume is received, namespace and volume are created, formatted with mkfs, mounted and NodePublishVolume returns OK. Same request is received few times with increasing time interval. e2e testing code which tries to create the pod, will fail with:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node storage
... and ownership changed for those volumes.


Question: allErrs = append(allErrs, apivalidation.ValidateNonnegativeField(int64(spec.Replicas), fldPath.Child("replicas"))...) allErrs = append(allErrs, apivalidation.ValidateNonnegativeField(int64(spec.MinReadySeconds), fldPath.Child("minReadySeconds"))...) <del> if utilfeature.DefaultFeatureGate.Enabled(features.StatefulSetStartOrdinal) { if spec.Ordinals != nil { replicaStartOrdinal := spec.Ordinals.Start allErrs = append(allErrs, apivalidation.ValidateNonnegativeField(int64(replicaStartOrdinal), fldPath.Child("ordinals.start"))...) } </del> <ins> if spec.Ordinals != nil { replicaStartOrdinal := spec.Ordinals.Start allErrs = append(allErrs, apivalidation.ValidateNonnegativeField(int64(replicaStartOrdinal), fldPath.Child("ordinals.start"))...) </ins> } if spec.Selector == nil {

This applies to StatefulSetStartOrdinal feature gate. Field validation should be done regardless of the feature gate state. From
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig apps


Question: library = ":go_default_library", tags = ["automanaged"], deps = [ <ins> "//vendor/golang.org/x/net/websocket:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library",

an APIService that routes to an external API server (e.g. service catalog) via a service in namespace a caBundle in the APIService the external API server with a certificate signed for Result: Normal API requests (get/put/post) work as expected Normal watch requests work as expected Upgrade requests (e.g. websocket watch requests) return a 500 error with text like cc


Question: "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" <del> "k8s.io/apimachinery/pkg/util/wait" </del> kmsapi "k8s.io/kms/apis/v2" )

Drop the and dependencies from to keep the import lighter. /sig auth /triage accepted
/assign Hi, , I created a PR to fix this. I am not sure If I am correct due to the repo seems generated. Would you like to provides some hints for me?
This will likely require a new go module in to move the client-go dep and then mark it as internal, ex: . is the only API machinery dep so we can probably come up with some way to re-implement the wait logic locally? Then we drop these lines so that these imports do not come back again:
Whoa, yeah, I'd take a for loop and a time.Sleep in a test to drop those deps
What is kms using client-go for? Tests only?
These were all needed by the KMS repo back when we had a much more complicated reference implementation which required a key hierarchy. Now they are just used in some tests and mock implementations, so we should be able to simplify things.
I found a place for rate limit. #liggitt
that package just delegates to "", switch that mock provider to use that directly?


Question: } func (server *APIServer) handleIndex(w http.ResponseWriter, req *http.Request) {  if req.URL.Path != "/" && req.URL.Path != "/index.html" { server.notFound(w, req) return }  w.WriteHeader(http.StatusOK) // TODO: serve this out of a file? data := "Welcome to Kubernetes"

sig-release master-blocking build-masterci-kubernetes-build.Overall 2022-06-01 20:49 CEST, commit No response /sig release cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm checking that
/assign
this pr should fix
it looks like it's still failing: https://k8s-
/reopen reopening as this is causing all our CI to run against stale artifacts at: v1.25.0-alpha.0.734+
Reopened this issue. <detailsIn response to : /reopen v1.25.0-alpha.0.734+ </details
my previous fix only fixed one side of the things, missing now job are green ! closing for now /close
Closing this issue. <detailsIn response to : my previous fix only fixed one side of the things, missing now job are green /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thank you !


Question: 1. More intelligent defaulting of fields (e.g., [#2643](https://github.com/GoogleCloudPlatform/kubernetes/issues/2643)) 1. Update preconditions based on the values of arbitrary object fields. 1. Deployment manager compatibility on GCP: [#3685](https://github.com/GoogleCloudPlatform/kubernetes/issues/3685) <ins> 1. Describe multiple objects, multiple kinds of objects [#5905](https://github.com/GoogleCloudPlatform/kubernetes/issues/5905) 1. Support yaml document separator [#5840](https://github.com/GoogleCloudPlatform/kubernetes/issues/5840) </ins> TODO: * watch <del> * ssh [#1513](https://github.com/GoogleCloudPlatform/kubernetes/issues/1513) </del> * attach [#1521](https://github.com/GoogleCloudPlatform/kubernetes/issues/1521) * image/registry commands * do any other server paths make sense? validate? generic curl functionality?

Simple version is done, more complex version needs work. Done, now that readiness corresponds to Running and is tracked by Kubelet status if the readiness hook is set? Done Done Do we want to do a more generic one or a less generic one? OpenShift was looking more at specific topical plugin commands ("add-volume", "set-default-resources"), but not sure how much hand-holding users need. Getting close. Exec has landed and is a replacement for ssh Attach on start remains hard. Implemented
I/we should also file issues for outstanding items without issues, and add lines for newly filed issues (e.g., command flag for run-container). Readiness is not yet used by rollingupdate. With respect to patch on the command line: I was thinking of a generic mechanism, but I could see the case for domain-aware commands, also. I think we need to better factor the command code and library code, and generic functionality vs. domain-aware functionality. Neither should ever convert to the internal representation. Domain-aware functionality (e.g., describe) should target a specific API version. Configuration generators, such as run-container, should be provided as services, or at least as separate plugin commands.
ref
ref
ref
ref
ref
ref


Question: SecurityContext: &v1.SecurityContext{}, }, }, <del> NodeName: nodeName, </del> }, }, }, } <ins> if len(nodeName) > 0 { job.Spec.Template.Spec.NodeSelector = map[string]string{ "kubernetes.io/hostname": nodeName, } } </ins> switch behavior { case "notTerminate": job.Spec.Template.Spec.Containers[0].Command = []string{"sleep", "1000000"}

pull-kubernetes-e2e-kind example: Example, but does not seem to be specific to the test: "Using a pod failure policy to not count some failures towards the backoffLimit". First time noticed, but potentially for long if this is very rare. The 3rd pod for the job could not get assigned resources by kubelet to start. See kubelet logs from kind-worker2: Relevant lines: Due to hitting the limit the pod is marked as by kubelet which is unexpected by the test. No response /sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I suppose the reason is that we preassigning the instead of asking scheduler to do so. This might be related to
FYI
we should not preassign, is that required by this test? Do you have a link to those tests?
What is required is that all pods run on the same node, because we use HostPath to communicate some information between recreations of a pod, and between replicas for indexed job. All tests in this file: , which use the function are affected I think. Here is the problematic place: I suppose using a instead of setting directly might already help, because it the decision will go via scheduler which will delay the scheduling until node has enough capacity.
FYI for input on the proposed solution to use or another alternative
hmm ,I don't like why this is this way , if the test need to be serialized it must be tagged as Serial, but making this assumptions on e2e tests is not great
Yeah, I would very much want to avoid making the tests serial. I can see two options: term - replace the with . IIUC, this will already keep the pods unscheduled until there is capacity. With the timeout of 15min as we have for job it is very unlikely the pod would wait that long. Currently, it gets assigned but fails. the tests so that the state is kept on a mounted volume obtained by PVC.

This should eliminate the need for the pods to run on the same node. Since 2. is slightly more involving, it might be good to have 1. merged.
you can not assume the e2e test will have dynamic PV working, if you want to assume that you have to tag the test so jobs can filter based on the tag
Let's start with the NodeSelector
Actually, that wouldn't help, because currently the Job controller just recreates the Pod. With a node selector, it would just be scheduler that is now "busy-waiting" and we would still hit a timeout. Can the implementation be changed to save the information in a ConfigMap? It probably requires a specialized container image. Unless a ConfigMap can be mounted as readwrite?
Interesting, so I greped the logs for and only found this pod in this line: Oct 24 07:42:50 kind-worker2 kubelet[248]: I1024 07:42:50. 248 ] "Event occurred" object="job-712/pod-disruption-failure-ignore-3-mpvmm" fieldPath="" kind="Pod" apiVersion="v1" type="Warning" reason="OutOfpods" message="Node didn't have enough resource: pods, requested: 1, used: 110, capacity: 110" Which should not be that bad, because we have backoffLimit=6, yet the job failed with . So, maybe some other failures were incrementing the counter wrongly. I don't know at the moment.
My mistake here. The job has actually : This explains why the job failed with BackoffLimitExceeded after seeing failing with admission error by kubelet. Meaning 137 exit code, but without DisruptionTarget condition. So it was counted as failed and killed the job. So, I think in this case adding the would actually help. It would not help, if was for a prolonged amount of time to cause test timeout. However, the test timeout is 15min. Also, there is no indication that kubelet was in the time for long. A single event was enough to cause the pod fail with 137, and this single failure caused the job failure.

As for long term solution using config maps - yes, it could work, but I think it's better to do a quick fix to minimize the flakes before release.
I agree. Thanks for investigating!


Question: import ( "bytes" <ins> "errors" </ins> "fmt" "io" "io/ioutil" <ins> "strings" </ins> "testing" "time"

What happened: When using a yaml file for input, silently fails at the first missing resource, not returning any further results. What you expected to happen: Searching with cli arguments properly ignores missing resources and continues the query, eg: Would expect similar results with a file input, eg: How to reproduce it (as minimally and precisely as possible): With an input file of : the command fails at apparently the first missing resource: Note: that rearranging the inputs in (so that the missing resource is last) produces the "expected" output: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: local OS (e.g: ): fedora 30 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: x-ref:
/sig cli
/assign
Got around to reproducing it and can confirm that I'm getting the same behavior. I'm looking into fixing it now
thanks, any updates on this?
I've learned that in the case of a fake entry, kubectl stops sending API get requests altogether. Learned by running kubectl get -f --ignore-not-found --v=10, the result is shown below. Till now I've been looking into the cobra command framework but now I'm trying to find out where kubectl is making these API calls from(working on finding out where this connection is happening from Slack+debugging, I've asked a more detailed question here ). <img width="1440" alt="Screen Shot 2019-08-12 at 4 13 41 PM" src="https://user-"
that's an interesting find, definitely helpful. When I looked into this I got to the section of get's main run function, and it seems like it should be continuing on any errors (see ) but maybe the ResourceBuilder is actually stopping when it hits a 404?
This behavior comes from where the not found errors are being ignored and where the input list is flattened. Errors are ignored here: While flattening occurs here: As the visitor iterates through the objects in the list if an error occurs, such as not found, it will stop iterating over the objects in the list and return that not found error. The logic will then ignore it and continue.

Any objects that came before the not found object in the list will have been visited and will then be included in the output. To make the behavior consistent between the two ways of getting the set of objects, the would need to be updated to support filtering errors.
Just submitted a PR adding that functionality to the FlattenListVisitor. The behavior is now consistent between the two methods.


Question: return false } <ins> func hasJobCondition(conditions []batch.JobCondition, conditionType batch.JobConditionType) bool { for _, condition := range conditions { if condition.Type == conditionType { return condition.Status == api.ConditionTrue } } return false } </ins> func printPodTemplate(obj *api.PodTemplate, options printers.GenerateOptions) ([]metav1.TableRow, error) { row := metav1.TableRow{ Object: runtime.RawExtension{Object: obj},

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: return mountErr } <del> if !*volumeSource.ReadOnly { </del> <ins> if volumeSource.ReadOnly == nil || !*volumeSource.ReadOnly { </ins> volume.SetVolumeOwnership(m, fsGroup) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I'm running a Redis pod with a PV provisioned as an Azure Managed Disk. It's got an exec readiness probe that calls in the container. At some point it receives a SIGTERM. The kubelet then tries to sync the pod and restart the container, but it panics in (looking over the code, that's the only place where the error from is ignored) As a result, the pod hangs in a state, since the readiness probe cannot be ran in a non existing container. What you expected to happen: The container would successfully restart. How to reproduce it (as minimally and precisely as possible): It's the 2nd time it happens but I haven't yet figured out the conditions to reproduce. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: Azure w/ Managed Disks OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi could you also share the storageclass, pvc & pod yaml files? I would like to repro, thx
I may insert a nil check statment in this func: But I need to reproduce this bug anyway, so that I could find the root cause of this issue, thx.
Yes, an error check is what I'm hoping for :) I've used the chart. The PV/PVC were created manually, they're set to an empty storage class. I should mention that I've also noticed other pods being terminated/recreated at the time this one failed, however I didn't trace the cause of this yet.
thanks for the reporting, I may have found the root cause, just need a few more debuging time, thx.
GitHub didn't allow me to assign the following users: xiazhang. Note that only can be assigned. <detailsIn response to : /assign xiazhang Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign andyzhangx
- Thanks for the fix!


Question: }, "io.k8s.api.core.v1.PodAffinityTerm": { "description": "Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key u003ctopologyKeyu003e matches that of any node on which a pod of the set of pods is running", <ins> "required": [ "topologyKey" ], </ins> "properties": { "labelSelector": { "description": "A label query over a set of resources, in this case pods.",

The field is marked as optional: The comment also describes a use case for empty topologykey: But the validation rule says it's required:
cc
cc could you take a look? Thank you. The validation rule is by
the release note of should go into the "action-required" section of the , but it's currently in "Other notable changes".
/assign
It seems that we should also update the external api type comments. Right?
I will look into it today.
If you didn't work on this, I am pleasure to send PR to fix this. :)
yeah sure. thanks.


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/kubernetes/pkg/scheduler/apis/config" "k8s.io/kubernetes/pkg/scheduler/metrics" schedulernodeinfo "k8s.io/kubernetes/pkg/scheduler/nodeinfo"

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: n, err = t.reader.Read(p) if err != nil { if t.o.MaxTries < 0 || t.retries < t.o.MaxTries { <ins> t.retries++ </ins> fmt.Printf("Resuming copy at %d bytes, retry %d/%dn", t.bytesRead, t.retries, t.o.MaxTries) t.initReadFrom(t.bytesRead + 1) err = nil <del> t.retries++ </del> } else { fmt.Printf("Dropping out copy after %d retriesn", t.retries) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Trying to copy a large file from a container via kubectl cp with the new flag results in the following error: I executed the following command: EDIT: The command executes and the file is copied. However, the resulting file is somehow corrupted: However, the files have exactly the same size locally and on the pod: Also, the command exits with non-zero code and thus scripts may stop or abort at this point. What you expected to happen: The large file to be downloaded from the container completely and unharmed. Exit code 0 only if copy was successful How to reproduce it (as minimally and precisely as possible): Add a large file (mine was ~150M) to any location in the pod (preferrably a tar file). Execute kubectl cp to download the large file and use the flag. Try to untar the resulting file Anything else we need to know?: Related to Issue . I only experience this issue if I see kubectl doing at least one retry. Environment: Kubernetes version (use kubectl version): server v1.20.12, client v1.23.1 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. uname -a): Install tools: Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
tar: Removing leading `/' from member names so cancel first "/" from "" like this: k cp --retries=-1
Thanks for the hint. The warning is a and I was just ignoring it in my case. This issue is about the error message and the command exiting with a non-zero exit code.
I just updated the issue description, we see the tar files being corrupted after kubectl did at least one retry
/assign atiratree
Is it correct to start at position bytesRead+1? Or should this be bytesRead instead?

For example, if 1 byte was read (position 0), we would want to resume from position 1, not position 2. Just a thought, I haven't really looked at what initReadFrom does.
It should be correct since the byte start of tail (part of implementaion of initReadFrom) and bytes read match. Meaning that when we read 10 bytes we want to start reading 11th one. The actual problem was that we were running a wrong command when -1 retries is used: please see


Question: var errNotInVMSet = errors.New("vm is not in the vmset") var providerIDRE = regexp.MustCompile(`^` + CloudProviderName + `://(?:.*)/Microsoft.Compute/virtualMachines/(.+)$`) <del> // returns the full identifier of a machine func (az *Cloud) getMachineID(machineName string) string { </del> <ins> // getStandardMachineID returns the full identifier of a virtual machine. func (az *Cloud) getStandardMachineID(machineName string) string { </ins> return fmt.Sprintf( machineIDTemplate, az.SubscriptionID,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Question: } return currCmd } <ins> // Tests that the CancelRequest function is propogated to the wrapped Delegate // RoundTripper; but only if the Delegate implements the CancelRequest function. func TestCancelRequest(t *testing.T) { tests := map[string]struct { delegate  http.RoundTripper cancelled bool }{ "CancelRequest propagated to delegate": { delegate:  &cancellableRoundTripper{}, cancelled: true, }, "CancelRequest not propagated to delegate": { delegate:  &nonCancellableRoundTripper{}, cancelled: false, }, } for name, tc := range tests { t.Run(name, func(t *testing.T) { rt := &CommandHeaderRoundTripper{ Delegate: tc.delegate, } req := http.Request{} rt.CancelRequest(&req) if tc.cancelled != req.Close { t.Errorf("expected RoundTripper cancel (%v), got (%v)", tc.cancelled, req.Close) } }) } } // Test RoundTripper with CancelRequest function. type cancellableRoundTripper struct{} func (rtc *cancellableRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { return nil, nil } func (rtc *cancellableRoundTripper) CancelRequest(req *http.Request) { req.Close = true } // Test RoundTripper without CancelRequest function. type nonCancellableRoundTripper struct{} func (rtc *nonCancellableRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { return nil, nil } </ins>

No errors printed. Not sure. This is the same thing as , was fixed by Happened in a CI build Kubernetes version (use ): kubectl version 1.22.0 from image. Cloud provider or hardware configuration: OS (e.g: ): linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cli
/triage accepted /assign
We'll need to cherry pick this to 1.22.
Looks like the cancellation was indeed from a timeout. Sean will have a fix but do you have more insight into the timeout?
The build I linked above is using a feature of GitLab Kubernetes Agent called , which allows to access Kubernetes API from a CI job. An agent is proxying requests into the cluster. When I was investigating timeouts I discovered that they happen because of in the agent. Throttling happened because kubectl was making way too many requests and the code was not caching the authentication results. After fixing this, I started to investigate why kubectl is making so many requests in the first place and discovered that the image that was used () has a read-only file system and discovery client from client-go cannot persist the discovery information to disc and hence it performs discovery 3 (!) times for a simple request. This results in 125 API calls. I tried to fix that in by caching the discovery client, but it doesn't help. The solution is to cache discovery info in memory because even if a single discovery client is used, it just discards the data and re-fetches it if it's not on disk. Please have a look at that PR.
The code that reported the error is The interface only has one function: The as a function of a appears to be a convention in most (but not all) Kubernetes .
For more context: the is implemented in the . This error message is printed when this cached discovery client encounters a timeout, and tries to cancel the request. The is wrapped by the , and it it unable to propagate the because the does not implement .
I'm experiencing this today, seeing this error on my two master nodes. The whole cluster is down. I was performing activities which were using the hard drive quite a bit. Not sure if that helps with debugging. Might be time to reboot the system hosting the node vms.

But I think I'll experiment some more being that it is a dev cluster. A reboot of each of the master nodes and things are coming back up. Worker nodes were fine.


Question: client, err := f.Client(cmd) checkErr(err) <del> rc, err := client.ReplicationControllers(namespace).Get(args[0]) checkErr(err) </del> generatorName := GetFlagString(cmd, "generator") generator, found := kubectl.Generators[generatorName] if !found {

expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" get se hostnames -o yaml apiVersion: v1beta1 containerPort: "9376" ... obviously, endpoints controller is not happy.


Question: } for _, addr := range addrs { <del> if addr.Type == v1.NodeInternalIP { </del> <ins> isIPv6 := net.ParseIP(addr.Address).To4() == nil if (addr.Type == v1.NodeInternalIP) && (isIPv6 == needIPv6) { </ins> return addr.Address, nil } } <del> return addrs[0].Address, nil </del> <ins> for _, addr := range addrs { isIPv6 := net.ParseIP(addr.Address).To4() == nil if (addr.Type == v1.NodeExternalIP) && (isIPv6 == needIPv6) { return addr.Address, nil } } // It should never return an address from a different IP Address family than the one needed return "", ErrNoAddressFound </ins> } // getAttachedInterfacesByID returns the node interfaces of the specified instance.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The Kubernetes master created routes in the neutron router that assign an IPv6 nexthop for a IPv4 prefix. This is not possible and causes a reachability problem between the pods. Example route: What you expected to happen: When describing a node the master has the following information It should detect the address type and choose the correct InternalIP How to reproduce it (as minimally and precisely as possible): Deploy on a Openstack cloud with IPv6 enabled using the following playbook Anything else we need to know?: The wrong next hop is coming from calling the function and nowhere is a check if the address is IPv4 or IPv6 Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ):Install tools: Others: /sig openstack
- Thank you for this detailed analysis. Seems to me that the best way to address this would be to add an IP family argument to getAddressByName() to let it know what V4/V6 family to look for for a next-hop address. In the calls to getAddressByName(), this argument could be set based on the family of route.DestinationCIDR.
Issue is probably related.


Question: s.config.Modeler.LockedAction(func() { bindingStart := time.Now() err := s.config.Binder.Bind(b) <del> metrics.BindingLatency.Observe(metrics.SinceInMicroseconds(bindingStart)) </del> if err != nil { glog.V(1).Infof("Failed to bind pod: %+v", err) s.config.Recorder.Eventf(pod, api.EventTypeNormal, "FailedScheduling", "Binding rejected: %v", err) s.config.Error(pod, err) return } <ins> metrics.BindingLatency.Observe(metrics.SinceInMicroseconds(bindingStart)) </ins> s.config.Recorder.Eventf(pod, api.EventTypeNormal, "Scheduled", "Successfully assigned %v to %v", pod.Name, dest) // tell the model to assume that this binding took effect. assumed := *pod assumed.Spec.NodeName = dest s.config.Modeler.AssumePod(&assumed) }) <ins> metrics.E2eSchedulingLatency.Observe(metrics.SinceInMicroseconds(start)) </ins> }

Currently scheduler latency are observed regardless of failure. For example, binding latency is like: This could be a potential confusion if latency spikes happen because of failure. We should better handle it like separating error case from normal
/cc
I completely agree - are you going to change this?
Yes. I will do it.


Question: return 0 < port && port < 65536 } <ins> // Now in libcontainer UID/GID limits is 0 ~ 1<<31 - 1 // TODO: once we have a type for UID/GID we should make these that type. const ( minUserID  = 0 maxUserID  = math.MaxInt32 minGroupID = 0 maxGroupID = math.MaxInt32 ) // IsValidGroupId tests that the argument is a valid gids. func IsValidGroupId(gid int64) bool { return minGroupID <= gid && gid <= maxGroupID } // IsValidUserId tests that the argument is a valid uids. func IsValidUserId(uid int64) bool { return minUserID <= uid && uid <= maxUserID } </ins> const doubleHyphensFmt string = ".*(--).*" var doubleHyphensRegexp = regexp.MustCompile("^" + doubleHyphensFmt + "$")

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ Local: &v1.LocalVolumeSource{ <ins> // Not needed because we don't need to detach local device from the host. </ins> Path: "", }, },

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (block volmode)] disruptive[Disruptive] Should test that pv used in a pod that is force deleted while the kubelet is down cleans up when the kubelet returns. Since when has it been failing: 10/8/19 Testgrid link: https://k8s- Reason for failure: Introduced by but root cause is probably Anything else we need to know:
/assign
/milestone 1.17 /priority important-soon this is consistently failing in release-master-informing job could you please take a look?
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.17 /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17


Question: w.Write(LEVEL_2, "NodeName:t%sn", nodeNameText) zoneText := "<unset>" <del> if endpoint.NodeName != nil { </del> <ins> if endpoint.Zone != nil { </ins> zoneText = *endpoint.Zone } w.Write(LEVEL_2, "Zone:t%sn", zoneText)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --panic as below when an Endpoint doesn't have zone. should print zone as if an Endpoint's zone is unset. not set "" for Nodes. Pods and export them as a service. kubectl to describe the corresponding EndpointSlice. Kubernetes version (use ): v1.21.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cli /assign


Question: verbs:     "get,watch,list", resources: "pods,deployments.extensions", expectedRole: &rbac.Role{ <ins> TypeMeta: v1.TypeMeta{APIVersion: "rbac.authorization.k8s.io/v1", Kind: "Role"}, </ins> ObjectMeta: v1.ObjectMeta{ Name: roleName, },


Question: if len(containers) > 0 { firstContainer, containers = containers[0], containers[1:] } <del> _, err := fmt.Fprintf(w, "%st%st%st%st%st%sn", </del> <ins> _, err := fmt.Fprintf(w, "%s/%st%st%st%st%st%sn", </ins> pod.Name, <ins> pod.Status.PodIP, </ins> firstContainer.Name, firstContainer.Image, podHostString(pod.Status.Host, pod.Status.HostIP),

From IRC ... Currently kubectl get pod <idonly list Host IP on which the pod bounds to, there is no PodIP information. The only way for the user to know PodIP is run the same command with "-o json" flag. This is confusion to the user. cc
Already working on a PR for this.
Yay.


Question: if err != nil { glog.Errorf("[cpumanager] reconcileState: failed to add container (pod: %s, container: %s, container id: %s, error: %v)", pod.Name, container.Name, containerID, err) failure = append(failure, reconciledContainer{pod.Name, container.Name, containerID}) <ins> continue </ins> } } else { // if DeletionTimestamp is set, pod has already been removed from state

/kind bug What happened: Sometimes when an deployment creates new pods they get scheduled on a node successfully but the containers fail to start with: What you expected to happen: The scheduler shouldn't put pods on nodes without enough resources. How to reproduce it (as minimally and precisely as possible): I don't know how it got into this situation. Deleting the pods and have the RS recreate them works around this issue. Anything else we need to know?: We're using the CPUManager with static policy which might contribute to this problem. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): coreos /sig scheduling
/sig node (for CPU manager)
do the Kubelet logs seem similar to what's reported in this PR description?
/assign
Hi! :) No, I don't see these 'still running' messages. This is what I've got in the kubelet log (slightly redacted):
Can you share more information on this? Deployment manifest, platform details, frequency of these errors?
Fortunately we have open source the platform deployment: - but in short it's: AWS (c5.4xlarge instances for the workers where this issue is happening) multi master w/ kubeadm Enabled CPUManager and set to static policy Running 5 ASG: Controller, GPU-enabled workers and 3x regular workers (one per AZ) Using cluster autoscaler and HPA We have this problem with some pods pretty much all the time. Since we have pretty varying workloads, the CA often moves pods around and scales up/down which works around the problem by deleting the affected pods. So sometimes all pods are healthy, sometime maybe 15% of the pods have this issue. The biggest issue for us currently is that it's blocking rolling upgrades: Since new pods take forever to start, it blocks our CI. There are several deployments affected. All use HPA and all have resource limits of 2 cpus. One has 5Gi, the other 2Gi memory limits but that shouldn't matter. Here is one deployment manifest (slightly edited for internals) Why does this "Internal PreStartContainer hook" even exists? Shouldn't the scheduler be what knows about available resources and decides about this? Is there some doc that could help me understand how this is suppose to work?

I'd imaging if there is some limits the scheduler doesn't know about (like exclusively available cores or something), it would explain the issue but it would be unclear how this can work at all?
I've just realize that only one specific node (which is suppose to be the same as all other nodes) seems to be affected on which all pods that require full CPUs (triggering the static allocation) fail. Going to taint and drain the node now and will decomision it ~tomorrow, so let me know if you want me to pull something from the node first. Unfortunately I don't have long enough log rentention to see what kubelet might have logged before it got into this state.
My kubelet args:
Maybe relevant: Now that I'm trying to drain the node, I realize that there are ~100 argo () workflows objects "mapped" to the node. I don't know the details and how they are related to the node, I only know that I can't drain the node because of the workflows (that's another issue with argo or CRDs in general, see ). Maybe this is somehow blocking the cpus?
Ah, I misunderstood that error. There are ~100 completed pods that were created by argo. Could it by simply that somehow the finished pods don't free their cpu resources somehow?
We have fixed the completed pods related issue before:
would you mind digging into the cpu manager state checkpointed to disk? This will confirm or refute for us that we failed to release CPUs from a terminal container. The state file is located by default in and the content is simply utf-8 JSON. We should be able to see some container-to-cpuset mappings in there. If we can verify that some of the mappings are for terminal container IDs then that's where we should focus the search. My next question would be, were the PreStopContainer and PostStopContainer callbacks invoked, and if so, was there a failed write to the checkpoint file. PreStopContainer is not guaranteed to be called, in case the container process exits on its own. PostStopContainer is called as part of container GC. If there's some other problem interfering with the GC logic then that may have prevented cleanup of our CPU manager state too. Thanks for helping us to debug this remotely!

Ah damn, looks like the cluster autoscaler terminated that node already so unfortunately can't provide the cpumanagerstate.. Would be nice to have the relevant bits exposed as Prometheus metrics. I'll update this issue if this happens again.
"Good" news, problem just re-appeared - this time on two nodes. Here is one: cpumanagerstate: One of the containers doesn't exist at all and another one is exited: Here is the state field from for the exited container: It has "": "koko-intelligence--np2zh", set and according to k8s this pod is still running on that host: I found this in the kubelet log for this container:
Oh and it's indeed running, now in container which is also properly accounted for in the state file. Now regarding the non-existent container, I also see it in the kubelet log: Looks like this belongs to koko-gateway-track--6mljm, which doesn't exist anymore. But most likely because it just got gracefully removed eventually. Maybe some race condition preventing it to remove it and leading to these inconsistencies?
This is awesome info — thanks for updating. I won’t have time to look at this immediately, but perhaps can help trace it. I suspect some kind of race in reconcileState.
Thanks for the info I will take a look at this. /assign
/ Do you have a rough estimate for this? Fully understand if you have other things on your plate but then I'd probably revert the static policy since this becomes a bigger and bigger problem for us (it looks like it got worse over time, now even nodes are affected that were just brought up).
/ Is there some way I can help you debug/fix this? Seems unrelated to anything special in our setup from what I can tell. We had to disable this for all our deployments because it was causing severe issues.
/ Do you have any update/ETA on this?
Note that this is fixed in 1.12, but backports to 1.11 and 1.10 are not merged yet:
This still happens occasionally in 1.12 (with ) . provides cleanup for stalled entries.
Out of curiousity, does anyone still seeing this issue utilize containers in their pod Specs?

If so, the curlprit is likely addressed by the following:
The issue is still existing in kubernetes v1.14.4, without init containers to pod spec. Some way below may reproduce it: container crash on a node frequently and then delete the pod. a new pod and schedule to this node. Is this already fixed on master branch?


Question: // ref: #101697, #94626 - ImagePullSecrets are allowed to have empty secret names // which would fail the 2-way merge patch generation on Pod patches // due to the mergeKey being the name field <del> name: "Should update pod conditions successfully on a pod Spec with secrets with empty name", </del> <ins> name:   "Should update pod conditions successfully on a pod Spec with secrets with empty name", client: clientsetfake.NewSimpleClientset(), </ins> pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "ns", <del> Name:      "pod2", </del> <ins> Name:      "pod1", </ins> }, Spec: v1.PodSpec{ // this will serialize to imagePullSecrets:[{}]

If there is a connection failure when updating the Pod status, we don't retry. Furthermore, due to , we won't have another chance until much later. The problem is that if we don't mark the Pod as Unschedulable, other controllers (such as cluster-autoscaler) wouldn't react to these pods. We should have stronger guarantees to mark a Pod as Unschedulable. You need an unschedulable Pod (for example, a Pod with a non matching node affinity) and a flaky connection to apiserver. No response master This is particularly problematic in 1.24, due to . Any <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
That is bad, retrying in place on transient errors should probably be ok.
I'll address it. /assign
Thanks!


Question: go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw= go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= <ins> go.uber.org/automaxprocs v1.5.2 h1:2LxUOGiR3O6tw8ui5sZa2LAaHnsviZdVOUZw4fvbnME= go.uber.org/automaxprocs v1.5.2/go.mod h1:eRbA25aqJrxAbsLO0xy5jVwPt7FQnRgjW+efnwa1WM0= </ins> go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= go.uber.org/multierr v1.6.0 h1:y6IPFStTAIT5Ytl7/XYmHvzXQ7S3g/IeZW9hyZ5thw4= go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=

<!-- Please only use this template for submitting enhancement requests --What would you like to be : should detect the appropriate GOMAXPROCS based off of the cpu limit supplied to the pod. Why is this needed: If not, test failures can occur (e.g. ) where GOMAXPROCS defaults to the number of cpus on the machine, while the cpu limit is actually far less. Too many compile/test processes launch, and any timing tests can become flaky. The end result is that running the test jobs on a node with more CPUs can make it flaky when running the test on a node with fewer CPUs cleanly passes. It's possible to set GOMAXPROCS=cpu limits in test jobs to avoid this, but it would be nicer if would autodiscover and set GOMAXPROCS appropriately.
/sig testing /sig k8s-infra
/transfer kubernetes
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign dims
xref:
This is already done, which is great, but for some more info I did some benchmarking of setting GOMAXPROCs vs not: Pretty substantially better in some cases


Question: t.Fatalf("%d:, error writing to stdout: %v", i, err) } out.Close() <del> select { case <-clientStdoutReadDone: case <-time.After(10 * time.Millisecond): t.Fatalf("%d: timed out waiting for client to read stdout", i) } </del> <ins> <-clientStdoutReadDone </ins> } else if out != nil { t.Fatalf("%d: stdout: expected nil: %#v", i, out) }

this is you
Hmm, I had upped waiting from 100ms to 500ms and that seemed to fix it. Do you have any better ideas?
Don't wait? :)
If the only reason you would timeout would be the code is broken, that's why the test runs have -timeout set.
Then the test fails immediately. It's waiting for the net.Listen calls to finish so it can connect to the specified local ports.
*finish enough so that socket connections work
So I can remove the timeout gating in my tests?
Yes ----- Original Message -----
Just failed again: https://travis-
Seems to be failing frequently now.
If this is test is failing again, my gut says it's not the same root cause. The original failure was a timeout issue. This new one is connection reset by peer. I've got the test running in a loop waiting for a failure but haven't seen one yet. is this still failing frequently?
No. It had failed twice in ~10 builds when I reopened this issue, but hasn't failed since. Reclosing. Will file a new issue if it recurs.


Question: return admission.NewForbidden(a, err) } <del> if d.hostPID && pod.Spec.SecurityContext != nil && pod.Spec.SecurityContext.HostPID { return admission.NewForbidden(a, fmt.Errorf("cannot exec into or attach to a container using host pid")) } </del> <ins> if pod.Spec.SecurityContext != nil { securityContext := pod.Spec.SecurityContext if d.hostNetwork && securityContext.HostNetwork { return admission.NewForbidden(a, fmt.Errorf("cannot exec into or attach to a container using host network")) } </ins> <del> if d.hostIPC && pod.Spec.SecurityContext != nil && pod.Spec.SecurityContext.HostIPC { return admission.NewForbidden(a, fmt.Errorf("cannot exec into or attach to a container using host ipc")) </del> <ins> if d.hostPID && securityContext.HostPID { return admission.NewForbidden(a, fmt.Errorf("cannot exec into or attach to a container using host pid")) } if d.hostIPC && securityContext.HostIPC { return admission.NewForbidden(a, fmt.Errorf("cannot exec into or attach to a container using host ipc")) } </ins> } if d.privileged && isPrivileged(pod) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: currently only are checked in admission controller What you expected to happen: hostNetwork should also be checked to deny exec /attach How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind bug /sig auth
cc


Question: corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" <ins> apimachineryvalidation "k8s.io/apimachinery/pkg/api/validation" </ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" ) <del> const totalAnnotationSizeLimitB int64 = 256 * (1 << 10) // 256 kB </del> type lastAppliedUpdater struct { fieldManager Manager }

Expose totalAnnotationSizeLimitB in the validation package: This value is useful for validating annotations outside of the validation package, such as in SSA: Otherwise, size validation on annotations outside of the package would diverge and cause issues later. /kind cleanup /wg api-expression cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: } } <ins> for _, container := range pod.Spec.InitContainers { if container.Resources.Limits.Cpu().IsZero() { cpuLimitsDeclared = false } if container.Resources.Limits.Memory().IsZero() { memoryLimitsDeclared = false } containerHugePageLimits := HugePageLimits(container.Resources.Requests) for k, v := range containerHugePageLimits { if value, exists := hugePageLimits[k]; !exists || v > value { hugePageLimits[k] = v } } } </ins> // quota is not capped when cfs quota is disabled if !enforceCPULimits { cpuQuota = int64(-1)

I noticed ResourceConfigForPod isn't checking here: The methods called by ( and ) do check , but it looks like and can be true for burstable pods even when some of the initContainers do not declare cpu/memory limits. I'm not sure what the implications of setting these values are if an initContainer has no cpu or memory limit: /sig node /assign
After reading the code, I prefer that this is a bug according to . would you confirm? /assign I opened .
/triage accepted


Question: func init() { utilruntime.Must(logsapi.AddFeatureGates(utilfeature.DefaultMutableFeatureGate)) <ins> // Prevent memory leak from OTel metrics, which we don't use: // https://github.com/open-telemetry/opentelemetry-go-contrib/issues/5190 otel.SetMeterProvider(noop.NewMeterProvider()) </ins> } const (

Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29.

At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks:


Question: ErrCodeResourceVersionConflicts: "resource version conflicts", ErrCodeInvalidObj:               "invalid object", ErrCodeUnreachable:              "server unreachable", <ins> ErrCodeTimeout:                  "request timeout", </ins> } func NewKeyNotFoundError(key string, rv int64) *StorageError {

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "description": "When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history.", "required": false, "allowMultiple": false <del> }, { "type": "string", "paramType": "path", "name": "namespace", "description": "object name and auth scope, such as for teams and projects", "required": true, "allowMultiple": false </del> } ], "responseMessages": [

The selflink provided in the api response for component statuses (/api/v1/componentstatuses) is broken. For example: /api/v1/namespaces/componentstatuses/controller-manager (as specified in the response of /api/v1/componentstatuses), will return: { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } while: /api/v1/namespaces/default/componentstatuses/controller-manager, will return the data. In addition, and this is asked for better understanding, are the component statuses namespace scoped? as far as I understood etcd, the scheduler and the controller manager were shared across the kubernetes cluster. If they aren't namespace scoped, what is the logic behind specifing a namespace in these api requests (as shown in ): /api/v1/namespaces/{namespace}/componentstatuses /api/v1/namespaces/{namespace}/componentstatuses/{name} If they are in fact namespace scoped, would you be kind and share an explanation/link to a documentation page?
I agree that it doesn't make a lot of sense for this to be namespaced given what's in it at the moment. Given what I'd like it to contain, the namespace might make more sense in the future. The self link is super broken right now, though, probably due to lack of actual namespace. there's probably a bug in the go-restful setup code for this. Can you take a look at some point? Also it might be worth mentioning that this will probably change in the near future, is working on a replacement.
The whole /componentstatus API is just broken. It needs rewriting.