Question: } for _, tc := range tests { <del> fn := func(rule string) *apiextensions.CustomResourceDefinition { </del> <ins> fn := func(rule, messageExpression string) *apiextensions.CustomResourceDefinition { </ins> validationRules := []apiextensions.ValidationRule{ { <del> Rule: rule, </del> <ins> Rule:              rule, MessageExpression: messageExpression, </ins> }, } return &apiextensions.CustomResourceDefinition{

Noticed a copy & paste issue when reading through the CEL code. Basically here: This should be instead of: The impact of this issue should be that messageExpressions are always evaluated with the CEL environment of the compatibility version. But stored (i.e. pre-existing) messageExpressions should be evaluated with a CEL environment with the "max" version instead. For users of Kubernetes this means that messageExpressions deployed with one minor version of Kubernetes might not work anymore if they rollback e.g. by one minor version. messageExpressions should be treated the same way as rules in that regard. Deploy Kubernetes 1.30 Deploy CRD that uses messageExpression that is using Sets (which became available with Kubernetes 1.30 that is using compatibilty version 1.29) Rollback to Kubernetes 1.29 Update something in the CRD =CRD update will fail because messageExpression is evaluated with the wrong CEL environment (it should still work) No response <detailsAt least on Kubernetes main, but I would assume this isuse is there since CEL was implemented </details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Will open a PR in a bit


Question: // call this once you're sure the lease has been created func (c *controller) retryUpdateLease(ctx context.Context, base *coordinationv1.Lease) error { for i := 0; i < maxUpdateRetries; i++ { <del> leaseToUpdate, _ := c.newLease(base) lease, err := c.leaseClient.Update(ctx, leaseToUpdate, metav1.UpdateOptions{}) if err == nil { c.latestLease = lease return nil } klog.FromContext(ctx).Error(err, "Failed to update lease") // OptimisticLockError requires getting the newer version of lease to proceed. if apierrors.IsConflict(err) { base, _ = c.backoffEnsureLease(ctx) continue </del> <ins> leaseToUpdate, err := c.newLease(base) if err != nil { klog.FromContext(ctx).Error(err, "Failed to prepare lease") } else { lease, err := c.leaseClient.Update(ctx, leaseToUpdate, metav1.UpdateOptions{}) if err == nil { c.latestLease = lease return nil } klog.FromContext(ctx).Error(err, "Failed to update lease") // OptimisticLockError requires getting the newer version of lease to proceed. if apierrors.IsConflict(err) { base, _ = c.backoffEnsureLease(ctx) continue } </ins> } if i > 0 && c.onRepeatedHeartbeatFailure != nil { c.onRepeatedHeartbeatFailure()

Some external controller deleted node object while kubelet was alive Due to ignoring errors in kubelet's lease logic, it created lease without ownerReference set Kubelet doesn't create lease without ownerReference set in any case, even pathological one Some external controller deletes node object Garbage collector properly deletes associated lease object Kubelet tries to update object and it fails on Conflict (saying that UID doesn't match): We go to branch, call backoffEnsureLease and assign result to base, but backoffEnsureLease may return potentially invalid object if ensureLease returned error (which happens when newLeasePostProcessFunc returns error, e.g. when node doesn't exist). We The next loop calls c.newLease on the incorrect base and ignores error in line 181 (so the second attempt to call newLeasePostProcessFunc which also fails with an error) We do UPDATE with incomplete (without effect of newLeasePostProcessFunc) object Given that leases allow CreateOnUpdate a new object, without ownerReference is created: No response <details21.10 </details<details</details<details</details<details</details<details</details<details</details
/sig node
/assign
/cc
/triage accepted need to understand what is the right behavior here
We are in trouble for this problem too. I think It’s exactly as said. Lease has the special update endpoint which can create the new lease objects not only update them. If it receives a request with a new lease object which has non-nil UID field, returns a conflict error. This situation can be occured in the function when a lease object is deleted. The lease object is deleted when the node resource is deleted in the result of GC. When received a conflict error in updating a lease object and the parent node object doesn't exist, , and continue next loop. In the next loop, the function returns the new lease object without ownerReference and UID. As a result, it is created unexpectively by the special update endpoint.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: if cert, key, err := certutil.GenerateSelfSignedCertKeyWithFixtures(publicAddress, alternateIPs, alternateDNS, s.ServerCert.FixtureDirectory); err != nil { return fmt.Errorf("unable to generate self signed cert: %v", err) <del> } else { </del> <ins> } else if len(keyCert.CertFile) > 0 && len(keyCert.KeyFile) > 0 { </ins> if err := certutil.WriteCert(keyCert.CertFile, cert); err != nil { return err } if err := certutil.WriteKey(keyCert.KeyFile, key); err != nil { return err } glog.Infof("Generated self-signed cert (%s, %s)", keyCert.CertFile, keyCert.KeyFile) <ins> } else { tlsCert, err := tls.X509KeyPair(cert, key) if err != nil { return fmt.Errorf("unable to generate self signed cert: %v", err) } s.ServerCert.GeneratedCert = &tlsCert glog.Infof("Generated self-signed cert in-memory") </ins> } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk.

I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security).

Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?


Question: return antiAffinity.CalculateAntiAffinityPriority } <ins> // Classifies nodes into ones with labels and without labels. func (s *ServiceAntiAffinity) getNodeClassificationByLabels(nodes []*v1.Node) (map[string]string, []string) { labeledNodes := map[string]string{} nonLabeledNodes := []string{} for _, node := range nodes { if labels.Set(node.Labels).Has(s.label) { label := labels.Set(node.Labels).Get(s.label) labeledNodes[node.Name] = label } else { nonLabeledNodes = append(nonLabeledNodes, node.Name) } } return labeledNodes, nonLabeledNodes } </ins> // CalculateAntiAffinityPriority spreads pods by minimizing the number of pods belonging to the same service // on machines with the same value for a particular label. // The label to be considered is provided to the struct (ServiceAntiAffinity).

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.


Question: return &Options{ config:      new(kubeproxyconfig.KubeProxyConfiguration), healthzPort: ports.ProxyHealthzPort, <ins> metricsPort: ports.ProxyStatusPort, </ins> scheme:      scheme.Scheme, codecs:      scheme.Codecs, CleanupIPVS: true,

What happened: From We used to allow using or in and . Now the validation on kubeproxyconfig disallow that (it only allows ): This seems to break backward compatibility for those two flags. Please comment if I'm missing something. cc How to reproduce it (as minimally and precisely as possible): Try with kube-proxy. Environment: Kubernetes version (use ):
/sig network /kind bug
Tried , the error is: and , the error is: So maybe we need to loosen and revert ?
Loosen the validation sounds good. was meant for loosen the validation as well (allowing not just ip, but also ip:port), but for some reason it doesn't work as intended. Would be great to fix that instead of revert :)
find the root cause of not working as expected. introduced the regression: I send a PR to fix the two issues we mentioned in this thread.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I agree There's also the issue of IPv6-only: Results in:
Thanks for your report. Do you mind opening a new issue regarding IPv6 since the original issue was fixed? Thanks.
Sure. Thanks
Please see:


Question: spec = specHandler.DefaultSpec() } <del> // Pass all parameters as volume labels for Portworx server-side processing. spec.VolumeLabels = p.options.Parameters </del> <ins> // Pass all parameters as volume labels for Portworx server-side processing if len(p.options.Parameters) > 0 { spec.VolumeLabels = p.options.Parameters } else { spec.VolumeLabels = make(map[string]string, 0) } </ins> // Update the requested size in the spec spec.Size = uint64(requestGiB * volutil.GIB)

BUG REPORT: Portworx volume plugin when a PVC is created using a StorageClass with empty parameters /kind bug What happened: Create a StorageClass using empty parameters and then create a PVC using that storage class. The PVC will remain in pending state. On inspecting the logs of the controller manager, we find that the portworx volume plugin is panicing. What you expected to happen: PVC should have been created successfully. How to reproduce it (as minimally and precisely as possible): Create following storage class and PVC Anything else we need to know?: Environment: N/A Kubernetes version (use ): 1.9.7, 1.10.1, 1.8.8 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
/sig storage


Question: ginkgo.It(t.name, func(ctx context.Context) { m.init(ctx, testParameters{ registerDriver: true, <del> podInfo:        test.oldPodInfoOnMount}) </del> <ins> podInfo:        &test.oldPodInfoOnMount}) </ins> ginkgo.DeferCleanup(m.cleanup) <del> waitUntilPodInfoInLog(ctx, m, test.oldPodInfoOnMount != nil && *test.oldPodInfoOnMount, false, 1) m.update(utils.PatchCSIOptions{PodInfo: test.newPodInfoOnMount}) waitUntilPodInfoInLog(ctx, m, test.newPodInfoOnMount != nil && *test.newPodInfoOnMount, false, 2) </del> <ins> waitUntilPodInfoInLog(ctx, m, test.oldPodInfoOnMount, false) m.update(utils.PatchCSIOptions{PodInfo: &test.newPodInfoOnMount}) waitUntilPodInfoInLog(ctx, m, test.newPodInfoOnMount, false) </ins> }) } }) }) <del> func waitUntilPodInfoInLog(ctx context.Context, m *mockDriverSetup, expectPodInfo, expectEphemeral bool, expectedNumNodePublish int) { </del> <ins> func waitUntilPodInfoInLog(ctx context.Context, m *mockDriverSetup, expectPodInfo, expectEphemeral bool) { </ins> var err error utils.WaitUntil(framework.Poll, framework.PodStartTimeout, func() bool {

1 ci-kubernetes-e2e-gci-gce 1 ci-cos-containerd-e2e-cos-gce in the master blocking test grid 1 ci-kubernetes-e2e-gci-gce-network-proxy-grpc 1 ci-kubernetes-e2e-gci-gce-proto Kubernetes e2e suite.[It] [sig-storage] CSI Mock workload info CSI PodInfoOnMount Update should be passed when update from false to true 12/14 code thaw after 1.29.0 release was merged at the time and is related to PodInfoOnMount. /cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
testCase: podInfoOnMount should be passed when update from false to true kubelet log: csi-mockplugin-0-mock log: the function in the test case is not working as expected. expect log at Line 2, but actual log is at Line 3.
is called twice for the same volume sometime even though the volume is already published successfully. Not only this issue, but also other issues like I wonder if it is a potential bug or a normal behavior. cc what do you think?
good catch. I think that any test that depends on exact index of NodePublish call will be flaky, sometimes Kubelet has to retry NodePublish. It would be better to check for NodePublish parameter changes. For example, test should check there were 1 or more NodePublish calls with no pod info and then 1 or more NodePublish calls with pod info afterwards.


Question: } // Flush and remove all of our chains. Flushing all chains before removing them also removes all links between chains first. <del> for _, ch := range iptablesChains { </del> <ins> for _, ch := range iptablesCleanupChains { </ins> if err := ipt.FlushChain(ch.table, ch.chain); err != nil { if !utiliptables.IsNotFoundError(err) { klog.Errorf("Error removing iptables rules in ipvs proxier: %v", err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /sig network What happened: Kube-proxy () deployed as daemonset with ipvs mode produce during startup following error: With verbosity -v5 the logs looks like this: Looks like proxier is trying to perform cleanup part: but failed. Such error is generated only once during startup. What you expected to happen: Error should not exist :) Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: none OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Install tools: custom script based on docker Others:
I've build kube-proxy with additional information: and the logs looks like this:
same issue
Looks like also fixed this one.
I'm on 1.12.0 and the issue is still there I suppose is included in 1.12?
Updated my kube to 1.12.3 and I got same
I'm still seeing this on v1.12.4 as well
same here on v1.13.2, throws this error when kube-proxy initializes. Running on CoreOS in ipvs mode:
Hello, I also encountered this problem in Kubernetes 1.13.3. The operating system is CentOS 7.6. The kernel version is: Linux linux-node4 3.10.0-957.el7.x86_64
Why is this closed? It looks like an unresolved issue. I have the same errors with 1.12.1.
same as me
same as me,1.12.3
same problem OS: Ubuntu 18 kernel: 4.15.0-46-generic Kubernetes: 1.13.4
same problem OS: CentOS 7.6 kernel: 3.10.0-957.el7.x86_64 Kubernetes: 1.13.4
I met this issue on Centos 7.6 with NetworkManager installed, it seems related to the network.
Same problem OS: CentOS 7.4 Kubernetes: 1.12.5 NetworkManager not installed.

Some of the nodes does not have this issue while others do
I am getting in 1.14.0 as well in some worker nodes.
same problem,how to solve it
same problem 1.13.2 ,how fix it?
i update kube-proxy 1.14.2, the error message is disappear
same problem OS: Debian 8 Kubernetes: v1.15.0 iptables v1.4.21 kube-proxy mode ipvs I write some test code like kube-proxy --cleanup and found line 7 is , and run , the error is The problem is we should flush KUBE-LOAD-BALANCER and KUBE-NODE-PORT before delete KUBE-MARK-MASQ , I can try fix it.


Question: - baseImportPath: "./vendor/k8s.io/kms/" allowedImports: <del> - k8s.io/apimachinery - k8s.io/klog </del> - k8s.io/kms - baseImportPath: "./vendor/k8s.io/endpointslice/"

Drop the and dependencies from to keep the import lighter. /sig auth /triage accepted
/assign Hi, , I created a PR to fix this. I am not sure If I am correct due to the repo seems generated. Would you like to provides some hints for me?
This will likely require a new go module in to move the client-go dep and then mark it as internal, ex: . is the only API machinery dep so we can probably come up with some way to re-implement the wait logic locally? Then we drop these lines so that these imports do not come back again:
Whoa, yeah, I'd take a for loop and a time.Sleep in a test to drop those deps
What is kms using client-go for? Tests only?
These were all needed by the KMS repo back when we had a much more complicated reference implementation which required a key hierarchy. Now they are just used in some tests and mock implementations, so we should be able to simplify things.
I found a place for rate limit. #liggitt
that package just delegates to "", switch that mock provider to use that directly?


Question: Protocol:         "tcp6", ShouldRaiseError: true, }, <del> { // IPv6 address must be put into brackets. This test reveals this. Hostname:         "::1", Protocol:         "tcp6", ShouldRaiseError: true, }, </del> } for i, testCase := range testCases {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When using IPv6 addresses, the kubelet passes the address to client-go, which does not correctly format the address for use in API requests. For example: Failed to list v1.Pod: Get https://2001:129::136:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dnode136&resourceVersi on=0: dial tcp: too many colons in address 2001:129::136:6443 What you expected to happen: The URI should have https://[2001:129::136]:6443, so that SplitHostPort() will correctly be able to separate the host and port components. How to reproduce it (as minimally and precisely as possible): Create cluster using IPv6 address (e.g. kubeadm init --apiserver-advertise-address=2001:129::136). When the kubelet starts up, it'll log this error. Anything else we need to know?: This is being handled as part of effort to support IPv6 in Kubernetes. Was found by a co-worker, I'll work on a commit to fix. Environment: Kubernetes version (use ): Using latest, via KubeAdm Cloud provider or hardware configuration*: bare-metal OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-81-generic Install tools: Using KubeAdm Others:
/area client-libraries /area kubelet /area ipv6
/assign
The point of failure is here: SplitHostPort will accept IPv4 and IPv6 addresses, but the latter must have square brackets around the IPv6 address.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery because client-go
Should we open this on the repository?

I see in the client-go repo read me that it states: Does that confirm that I should do a PR here?
One thing that I was wonder, was whether or not the hostname/address provided in the call to getListener, has been validated. If so, and it can only be a hostname, IPv4, or IPv6 address as a string, we could just check if there is a ":" in the name to determine IPv6. Otherwise, I could use net.ParseIP(), and net.To4() to determine if IPv6. I'll start working with the latter assumption, but please let me know, if the former could be used.
Also, I'm making the assumption that we'd want to make client-go be able to handle IPv6 addresses, rather than having each user of client-go (e.g. kubelet) from ensuring that the IP address is of the right form. IOW, have client-go add the needed [] around IPv6 addresses, rather than have kubelet do that. My thought is that there may be cases where we want just the IP address and not the modified IP. Does that sound OK?


Question: var ( apply_long = templates.LongDesc(i18n.T(` Apply a configuration to a resource by filename or stdin. <del> This resource will be created if it doesn't exist yet. </del> <ins> The resource name must be specified. This resource will be created if it doesn't exist yet. </ins> To use 'apply', always create the resource initially with either 'apply' or 'create --save-config'. JSON and YAML formats are accepted.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): 'job generateName' (also checked via ) Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report Kubernetes version (use ): ~ Environment: Cloud provider or hardware configuration: minikube 0.18.0 on Linux Install tools: kubectl What happened: I'm trying to create a Job resource with a auto-generated name: Applying this with kubectl gives: ~ Adding makes the job work, but obviously now with a fixed name. What you expected to happen: A job with the name 'tool-job-SOMETHING' is created, and a pod tool-job-SOMETHING-ELSE is used to execute the job. How to reproduce it (as minimally and precisely as possible):
will verify the existence of the resources before take action. If the resources do not exist, it will firstly create them. If use , the resource name is not yet generated when verify the existence of the resource. I will submit a PR to fix this. Currently, you can use instead of as a workaround.
Do We need to update the documentation of how kubectl should work with generated names as part of fixing this issue ? I'm not sure what the semantics is there, after all you can either generate more names with unique resources OR modify existing resources....?
It is really a little confused when use with . I think there is need to update the documentation.
with you would have clear intention: just create the resource. is a combination of and , it will fallback to create the resource if not found(use namespace and name to query the resource) or it will do 3-rd way patch. So basically I would not hope to change this behavior of , with generate name it's not easy to check the existence, I would suggest use to create the resource, thus you can use to manage the resource in the future.
Thanks for your clarification. I will send a PR to update the documentation.
Yup that was my hope as well; this isn't a bug b/c of the fact that apply doesn't work if you don't have a deterministic way to check. So let's just update docs and then close the issue .

From my point of view: the explanation for 'apply'-vs-'create' makes perfectly sense, and using 'create' in my case is also reasonable. Having documentation pointing to the differences between those, and having just a bit more documentation on the 'generateName' itself - or close to it - would be very much appreciated. I agree that with that explanation it is not a bug in apply. Thank you for the explanation :)
I'm confused about how generateName and name are supposed to be validated. On a v1.11.2 vanilla Kubernetes cluster, a Deployment with generateName works with "kubectl create" but fails with "kubectl apply". On a v1.10.5-gke.3 GKE cluster, the Deployment works with both create and apply. cc
I really have to argue that this resolution is doubly ineffective. First, the documentation does not make it clear that a nameGenerator cannot be used to create jobs via . It took a full day of work to make it here! Second, more importantly, it really disrupts the overall yaml workflow to forbid with nameGenerators. I understand that there is some complexity to handle in the first application, but I think it would be well worth it to ensure that 'apply' is works across all the reasonable use-cases, including creating new Jobs. If this means adding the ability to specify merge strategy in the generateName key, so be it.
I completely agree with here. To me this seems to be a solution that breaks the overall workflow.
any update?


Question: pod.ObjectMeta.Annotations = map[string]string{ kubelettypes.ConfigSourceAnnotationKey: kubelettypes.FileSource, } <del> podPriority := scheduling.SystemCriticalPriority pod.Spec.Priority = &podPriority </del> <ins> pod.Spec.PriorityClassName = systemCriticalPriorityName pod.Spec.Priority = &value </ins> gomega.Expect(kubelettypes.IsCriticalPod(pod)).To(gomega.BeTrue(), "pod should be a critical pod") } else {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: // The Write method can be invoked multiple times, so we use a // latency tracker that sums up the duration from each call. ResponseWriteTracker DurationTracker <ins> // DecodeTracker is used to track latency incurred inside the function // that takes an object returned from the underlying storage layer // (etcd) and performs decoding of the response object. // When called multiple times, the latency incurred inside to // decode func each time will be summed up. DecodeTracker DurationTracker </ins> } type latencyTrackersKeyType int

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: populate(g, nodes, pods, pvs, attachments) identifier := nodeidentifier.NewDefaultNodeIdentifier() <del> authz := NewAuthorizer(g, identifier, bootstrappolicy.NodeRules()).(*NodeAuthorizer) </del> <ins> authz := NewAuthorizer(g, identifier, bootstrappolicy.NodeRules()) </ins> node0 := &user.DefaultInfo{Name: "system:node:node0", Groups: []string{"system:nodes"}}

What happened: I checked if kubelet () has permissions for with SelfsubjectAccessReviews and I saw that it has. But when I checked its permissions with SelfsubjectRulesReviews it didn't show it. What you expected to happen: I expected that the API SelfsubjectRulesReviews will show that has permssions for . How to reproduce it (as minimally and precisely as possible): You can check it on the master or the worker. Test SelfsubjectAccessReviews: You will get answer that it has permissions: Test SelfsubjectRulesReviews: It won't show the permissions. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm /area/api /kind/bug /sig api-machinery
/sig api-machinery
This is a more clean display that show the bug: I think the problem is related to the fact that the file contain the kubelet certificate which is part of the group :
/sig auth
What is the SelfSubjectRulesReview response from the server ()? Note that not all authorizers support enumerating permissions, and the API object status includes an attribute which explicitly indicates the result is incomplete. See
It has Details:
I noticed that it also doesn't show that it has permissions to view and . It probably related to the Node authorizer: Because I assume that these rules doesn't appear in RBAC.
The node authorizer does not enumerate permissions, so it does not contribute to SelfSubjectRulesReview results
fixes setting correctly when the API is requested with a kubelet's credentials


Question: "node_lease.go", "pods.go", "privileged.go", <del> "projected.go", </del> <ins> "projected_combined.go", "projected_configmap.go", "projected_downwardapi.go", "projected_secret.go", </ins> "runtime.go", "secrets.go", "secrets_volume.go",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Running the non- conformance tests with produces unpredictable results, sometimes tests show up again with a on the end. This is due to the following tests having multiple tests with identical names: - - What you expected to happen: Tests should have unique names. How to reproduce it (as minimally and precisely as possible): run the conformance tests with a skip on and Anything else we need to know?: example here: https://k8s- /sig node /sig testing cc
er this is probably better /sig storage cc
/assign /assign
/area conformance
/milestone v1.13


Question: } } if !quiet { <del> printDeletion(info, out, shortOutput) </del> <ins> printDeletion(info, out, shortOutput, gracePeriod) </ins> } return nil })

IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy


Question: } }, ContainerGCPeriod, wait.NeverStop) <ins> stopChan := make(chan struct{}) defer close(stopChan) // when the high threshold is set to 100, stub the image GC manager if kl.kubeletConfiguration.ImageGCHighThresholdPercent == 100 { glog.V(2).Infof("ImageGCHighThresholdPercent is set 100, Disable image GC") go func() { stopChan <- struct{}{} }() } </ins> prevImageGCFailed := false go wait.Until(func() { if err := kl.imageManager.GarbageCollect(); err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The kubelet will GC the images on the node to free space when the used space reaching the percent ot the high-threhold(90% as default), but when the containers are not just managed by kubernetes(such as managed by swarm together) and not wished to be GCed by kubelet, I found we can not close the GC function by the flag of image-gc-high-threshold. What you expected to happen: add one switch to close the function of imageGC in specific senario: when the flag image-gc-high-threshold is set as 0, we should not GC the images on the node, just record one warning event to kube-apiserver Or add another switch flag to close the whole function of imageGC How to reproduce it (as minimally and precisely as possible): run kubernetes with several nodes, and test with making the inode all consumed of the nodes then the kubelet can triggle the GC to delete the oldest images on the node regardless the kind of them. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind feature
/sig node
What happens when --image-gc-low-threshold and --image-gc-high-threshold are both set to 100 ?
when both set to 100, image GC will still delete one image to release space as it always runs into the logic and delete image in logic of func
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: }, { Key:      "iteration", <del> Operator: metav1.LabelSelectorOpIn, </del> <ins> Operator: metav1.LabelSelectorOpNotIn, </ins> Values:   []string{d.Spec.Template.Labels["iteration"]}, }, }, }, }, }, <del> }, </del> <ins> } </ins> } <ins> d.Spec.Template.Spec.Affinity = affinity </ins> }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should handle load balancer cleanup finalizer for service [sig-network] Services should only allow access from service loadbalancer source ranges Since when has it been failing: 2019-10-19 Testgrid link: https://k8s- Reason for failure: and Anything else we need to know: /sig scalability /sig network could you triage?
It seems that new test was at that point: [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout My personal guess is that we started doing too much load-balancer-related stuff at the same time.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think this is my fault. should fix it. Is there some way to those tests to run against a PR? /remove-triage unresolved
At the moment the fastest way would be to merge your fix and wait for the next run tomorrow.
Reopening just to keep this tracking - will close once we have green run.
the fix helped for a two tests, but one of them is still failing: https://k8s-
I disagree - there are couple loadbalancing-related tests that are flaking with high flakiness ration - I think this PR didn't change anything. So my feeling is that is may still be something like:
Still, the conclusion is that unfortunately more work in needed to put us back in a good shape.
I looked a bit into it and the conclusions are: "Services should handle load balancer cleanup finalizer for service [Slow]" test is consistently failing in small clusters too with: IIUC, the reason is that if we downgrade load-balancer service to non-loadbalaced one, it takes some time to remove the ingress from Status, but it is failing immediately due to this: - can you please take a look? the apps test (and I think couple more) are failing due to incorrect timeouts - I will send out a fix for it soon
Actually, this one will be still failing as of now - the reason for it is described here:


Question: return ret }, SelfLinker:         meta.NewAccessor(), <del> ClusterScoped:      crd.Spec.Scope == apiextensions.ClusterScoped, </del> <ins> ClusterScoped:      clusterScoped, </ins> SelfLinkPathPrefix: selfLinkPrefix, }, ContextFunc: func(req *http.Request) apirequest.Context {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: got an error when trying to use a fieldSelector when watching a custom resource What you expected to happen: no error How to reproduce it (as minimally and precisely as possible): a CRD and then a CR for it. to watch with a fieldSelector, such as Anything else we need to know?: The customresource implementation has a defined, and it looks like it should support at least a field selector for and , but I'm guessing we need a bit more in place to make this work for . Environment: Kubernetes version (use ): HEAD (commit ) /sig api-machinery I can try to submit a PR to fix once I finish my current work item, if nobody picks this up sooner.
/area third-party-resource
cc
This is a more general form of . I don't have immediate plans to work on either of them, so you're welcome to take it.
sounds good. Do you want me to close this as a dupe?
I think there's a useful distinction between the two issues. If we support only , that would be enough to but this one tracks the idea of supporting arbitrary field selectors. By the way, if I recall correctly, when I applied the hack mentioned in the other thread to skip the step for CRDs, I think I saw that actually arbitrary field selectors worked too (not just ). That would mean enabling this might be easy on a technical level, although we may need to think through whether this could tie our hands if we ever wanted to add versioning support for CRD in the future. This issue is a better home for that discussion than the other one.
This would help a lot with debugging in prow. Today, the only thing we can do is bubble up information from the spec of a prowjob as labels and use a label selector.
What about just having 's just return the passed in label and value without any "conversion" for now?
I don't think we'd want to do it for all Unstructured, since there's no guarantee all Unstructured objects are CRD objects. However, it should be pretty easy to arrange for ConvertFieldLabel to be a no-op for CRD objects, which makes sense currently because CRD only allows one version for a given resource.

We just need to make sure this will not prevent us from doing more with CRD versioning in the future. My suspicion is that it should be fine, but I haven't investigated properly.
ok, I can put together a PR to make this change. any gut reactions?
PR:
I am sorry to comment on this closed thread. Currently, I am working on a CRD and needs to support arbitrary field selectors. I've tried to do it by learning from what you do for . But it always throws error: . So, I am wondering if I am doing something wrong or it only supports limited field selectors for CRD, which is stated . Here is my code snippet:
CRDs still only support namespace and name for field selectors
Any plan to support arbitrary field selector for CRDs?
Not currently

Follow a TPR an instance a single instance return 400. (watch a collection of TPR works) master and kubectl version: 1.7.1 CRD has the same issue. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I was able to reproduce this with CRD on master. The problem is that kubectl implements a watch on a single object as a watch on the collection with a field selector that filters by . However, field selectors don't work on CRD objects because they are stored as , which does not support : This method is used to transform the field selector from versioned to internal field names: As a test, I bypassed that check for Unstructured objects, and watches on single CRD objects started working. Given that CRD doesn't support multiple versions anyway, it seems feasible to support this with a properly-targeted override of specific to CRD objects. cc
If we could support name and namespace generally on all objects containing object meta, I think I'm in favor of the concept.
I think it's feasible to fix this for CRD at some point. However, given that TPR is already gone in master, I don't think it's worth fixing for TPR unless it's somehow a regression in 1.7. Do you have any objections to that?
Having CRD support is good enough for me and service catalog. Will you backport the fix for CRD to 1.7?
At this stage I wouldn't expect that since it isn't a regression (I don't think) for either extension mechanism.
cc
I am also getting the same error when i try to watch CRDs and filter on a specific field. Im a currently using the to watch for CRDs. On a related note, i noticed that the takes a . Is there a way to pass a label selector so that i can watch CRDs using labels?
Also is this issue slated for 1.8?

(For CRDs that is)
The general issue of supporting field selectors for CRD is tracked in . I don't think it's slated for 1.8. The methods on should take a , which can contain an optional .


Question: hostname:       hostname, nodeIP:         nodeIP, portMapper:     &listenPortOpener{}, <ins> recorder:       recorder, </ins> }, nil }

When kube-proxy fails to open a nodeport, it gives up setting up the rest of the iptables rules for the service. Leaving the service not usable. Cannot even use it with clusterIP. Kube-proxy only logs this error in . No useful information can be retrieved with kubectl. Need to record a event at least.
Ideally would the changes be removed and then reported back? Is there a preferred log or place to return the error?
the event maybe a better solution; as other port maybe success.
Need to surface the problem somewhere. An event seems okay
, there's and in ; should we reuse it or create new recorders for ?
Just reuse it. After all, it is within the same component. Are you going to submit a PR?
sure, my pleasure.
Great! Thanks! Let me know when the PR is ready.


Question: ginkgo.By("waiting for available Replicas") eventFound = false <del> ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) </del> <ins> ctx, cancel = context.WithTimeout(context.Background(), 120*time.Second) </ins> defer cancel() _, err = watchUntilWithoutRetry(ctx, retryWatcher, func(watchEvent watch.Event) (bool, error) { var rc *v1.ReplicationController

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Triage link for more jobs that are affected: Which test(s) are failing: Since when has it been failing: Since October 23 Testgrid link: Reason for failure: Anything else we need to know: Example spyglass link: /sig apps /cc /priority important-soon
/cc missed this in the main body
/assign /triage accepted
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-conformance-kind-ipv6-parallel, 2nd top flake for per /area conformance this is a conformance test /remove-kind failing-test /kind flake this is not failing continuously, but is instead flaking
/assign
I increased the timeout while waiting for available Replicas.
Test grid green since Merge 5 Nov 2020 /cc


Question: return curPlugin.GetInTreePluginName(), nil } } <del> return "", fmt.Errorf("could not find in-tree plugin name from persistent volume %v", pv) </del> <ins> return "", fmt.Errorf("could not find in-tree plugin name from persistent volume %s", pv.Name) </ins> } else if vol != nil { for _, curPlugin := range inTreePlugins { if curPlugin.CanSupportInline(vol) { return curPlugin.GetInTreePluginName(), nil } } <del> return "", fmt.Errorf("could not find in-tree plugin name from volume %v", vol) </del> <ins> return "", fmt.Errorf("could not find in-tree plugin name from volume %s", vol.Name) </ins> } else { return "", errors.New("both persistent volume and volume are nil") }

I noticed that kube-controller-manager is consuming visible amount of CPU in completely idle 5k node cluster (~7 cores/s) while for comparison kube-apiserver was using much less that time. I fetched pprof and noticed that we are burning significant in metrics.(attachDetachStateCollector).CollectWithStability:! Going further, most of the cpu is used for fmt.Errorf for formatting v1.Volume in line 160: which is never actually used by the caller We do not use significant cpu for preparing error messages that are then ignored. pprof cluster with some pods and volumes. No response_ <details26.1 </details<detailsgke, any? </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
A trivial fix would be actually to change GetInTreePluginNameFromSpec to write only PV/Volume names instead of entire specs in error message.
/assign In scheduler we use this function to check if we can translate the in-tree PV to CSI, I think we also only need the name of the pv. I'll create a pr for this one line change.
/sig storage Instrumentation ?


Question: package volume import ( <ins> "errors" </ins> "fmt" "net" "strings"

In the events , the PVC has been successfully resized, but a warning-level log is still printed. And by checking the logs of kube-controller-manager, can see the following output. Do not print events with warning level. To expand a a CSI PVC, and then use "kubectl describe pvc" to view its events. (I am using xsky CSI to expand pvc, but based on my understanding of Kubernetes code, this issue can occur with any CSI.) ! ! Based on Kubernetes v1.27.3, I some logs in the func located in . These logs printed all the plugins before expansion, and I noticed that was not among them. In my understanding, it is normal for CSI to be absent from the plugins here because CSI expansion is scheduled through external-resizer instead of kube-controller-manager. Therefore, there should not be any warning-level logs printed here either. v1.27.3 (In version 1.18, the issue can also be reproduced) <details</detailsLinux debian-hk 5.10.0-23-amd64 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux kind v0.20.0 <details</details<details</details
/sig storage
We can change the ExternalExpanding event from Warning to Info. are you interested in making this change?
/triage accepted
Yes, I will submit the pull request in the next few days.
Hi Have you submitted the PR? I have a bugfix branch tested on v1.29.1. I would like to cherry-pick it to master if this is not fixed yet or not started.
You can go do it. Thank you!


Question: }, time.Minute*8, time.Second*4).ShouldNot(gomega.HaveOccurred()) }) ginkgo.AfterEach(func() { <ins> defer func() { if framework.TestContext.PrepullImages { // The test may cause the prepulled images to be evicted, // prepull those images again to ensure this test not affect following tests. PrePullAllImages() } }() </ins> ginkgo.By("delete the static pod") err := deleteStaticPod(podPath, staticPodName, ns) gomega.Expect(err).ShouldNot(gomega.HaveOccurred())

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Several tests Eg. this run: Since when has it been failing: Unknown Testgrid link: Reason for failure: (happens to all the images). Pods in the e2e test suites use pull policy , , and it looks like the test during DiskPressure, resulting in kubelet removing some of the pre pulled images. Anything else we need to know: The runs after each test. This should also be the case in the test too.
/assign /sig node
These tests are now constantly failing due to missing images. This makes it hard to see if some of the stuff we do now, or have done in the last few weeks, breaks some of the tests. PR should hopefully fix this, but it is not in the 1.16 milestone, but has and . Should we get this merged into master so that we can see if if fixes the problem, or should we wait until code thaw? Any thoughts? /cc /cc /cc
/priority important-longterm
Thanks for all the work you've put into this ! I think we should add to the milestone
+1 to - thanks for your hard work :)
still has a lot of failing tests. What are the next steps? /reopen
Reopened this issue. <detailsIn response to : still has a lot of failing tests. What are the next steps? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it has over the few last weeks become quite a mess... Looks like fixed some of the issues about image eviction problems, but there are still a few left. I think I have a solution, so ill try to submit a ~PR in a few hours~ edit: will do it tomorrow. A lot of new tests started failing 8-28, and that may be related to , but I am not totally sure..
Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! ! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above).

Ill create a new issue about all the failing tests where we can track them each together. Closing this issue, as this was only filed for the flaking becuase of the evicted images. :smile: /close cc
Closing this issue. <detailsIn response to : Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // Populates desiredStateOfWorld cache with one volume/pod. <ins> // Calls Run() // Verifies there is are attach/mount/etc calls and no detach/unmount calls. func Test_Run_Positive_VolumeAttachAndMountMigrationEnabled(t *testing.T) { // Arrange intreeToCSITranslator := csitrans.New() node := &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: string(nodeName), }, Spec: v1.NodeSpec{}, Status: v1.NodeStatus{ VolumesAttached: []v1.AttachedVolume{ { Name:       v1.UniqueVolumeName(fmt.Sprintf("fake-plugin/%s", "pd.csi.storage.gke.io-fake-device1")), DevicePath: "fake/path", }, }, }, } volumePluginMgr, fakePlugin := volumetesting.GetTestKubeletVolumePluginMgrWithNode(t, node) dsw := cache.NewDesiredStateOfWorld(volumePluginMgr) asw := cache.NewActualStateOfWorld(nodeName, volumePluginMgr) kubeClient := createTestClient(v1.AttachedVolume{ Name:       v1.UniqueVolumeName(fmt.Sprintf("fake-plugin/%s", "pd.csi.storage.gke.io-fake-device1")), DevicePath: "fake/path", }) fakeRecorder := &record.FakeRecorder{} fakeHandler := volumetesting.NewBlockVolumePathHandler() oex := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator( kubeClient, volumePluginMgr, fakeRecorder, fakeHandler)) reconciler := NewReconciler( kubeClient, true, /* controllerAttachDetachEnabled */ reconcilerLoopSleepDuration, waitForAttachTimeout, nodeName, dsw, asw, hasAddedPods, oex, mount.NewFakeMounter(nil), hostutil.NewFakeHostUtil(nil), volumePluginMgr, kubeletPodsDir) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod1", UID:  "pod1uid", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: "volume-name", VolumeSource: v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: "fake-device1", }, }, }, }, }, } volumeSpec := &volume.Spec{Volume: &pod.Spec.Volumes[0]} migratedSpec, err := csimigration.TranslateInTreeSpecToCSI(volumeSpec, pod.Namespace, intreeToCSITranslator) if err != nil { t.Fatalf("unexpected error while translating spec %v: %v", volumeSpec, err) } podName := util.GetUniquePodName(pod) generatedVolumeName, err := dsw.AddPodToVolume( podName, pod, migratedSpec, migratedSpec.Name(), "" /* volumeGidValue */) // Assert if err != nil { t.Fatalf("AddPodToVolume failed. Expected: <no error> Actual: <%v>", err) } dsw.MarkVolumesReportedInUse([]v1.UniqueVolumeName{generatedVolumeName}) // Act runReconciler(reconciler) waitForMount(t, fakePlugin, generatedVolumeName, asw) // Assert assert.NoError(t, volumetesting.VerifyWaitForAttachCallCount( 1 /* expectedWaitForAttachCallCount */, fakePlugin)) assert.NoError(t, volumetesting.VerifyMountDeviceCallCount( 1 /* expectedMountDeviceCallCount */, fakePlugin)) assert.NoError(t, volumetesting.VerifySetUpCallCount( 1 /* expectedSetUpCallCount */, fakePlugin)) assert.NoError(t, volumetesting.VerifyZeroTearDownCallCount(fakePlugin)) assert.NoError(t, volumetesting.VerifyZeroDetachCallCount(fakePlugin)) } // Populates desiredStateOfWorld cache with one volume/pod. </ins> // Enables controllerAttachDetachEnabled. // Calls Run() // Verifies there is one mount call and no unmount calls.

The following test failures started happening on 30th March: Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (default fs)] volumes should allow exec of files on the volume Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (default fs)] volumes should store data Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (ext4)] volumes should allow exec of files on the volume Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (ext4)] volumes should store data See This seems related to the changes in kubelet fails with the following: Tests passing See the e2e logs above No response <detailsLatest CI build </details<detailsAWS </details<details</details<details</details<details</details<detailsEBS CSI driver 1.5.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig storage
/assign
not having debugging kops builds for awhile, are kubelet logs attached in prow? I don't see them...
Spoke with offline. It looks like we don't have kubelet logs captured in these tests. I need complete kubelet logs to make sense of these errors because testgrid shows inline volume tests as failing and code in question should not be called or inline volumes (because they can not be expanded). If that code fails, it would have been failing before too. I will wait for logs from and in the meanwhile try and see if I can reproduce these failures in single node AWS cluster setup that I can build manually.
thanks
Looks like this is happening for inline volumes if CSI migration is enabled for them. We construct a "fake" object on the fly for migration enabled plugins even though volume is inline, which is causing this condition to fail. It should be simple fix.
opened to fix the problem. The code change is just one line but has tests.


Question: // called from provisionClaim(), in this case, plugin MUST NOT be nil // NOTE: checks on plugin/storageClass has been saved pluginName := plugin.GetPluginName() <ins> if pluginName != "kubernetes.io/csi" && claim.Spec.DataSource != nil { // Only CSI plugin can have a DataSource. Fail the operation // if Datasource in Claim is not nil and it is not a CSI plugin, strerr := fmt.Sprintf("plugin %q is not a CSI plugin. Only CSI plugin can provision a claim with a datasource", pluginName) klog.V(2).Infof(strerr) ctrl.eventRecorder.Event(claim, v1.EventTypeWarning, events.ProvisioningFailed, strerr) return pluginName, fmt.Errorf(strerr) } </ins> provisionerName := storageClass.Provisioner <ins> klog.V(4).Infof("provisionClaimOperation [%s]: plugin name: %s, provisioner name: %s", claimToClaimKey(claim), pluginName, provisionerName) </ins> // Add provisioner annotation to be consistent with external provisioner workflow newClaim, err := ctrl.setClaimProvisioner(claim, provisionerName)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In clusters that have intree drivers as the default storageclass, if you try to create a PVC with snapshot data source and forget to put the csi storageclass in it, then an empty PVC will be provisioned using the default storageclass. What you expected to happen: PVC creation should not proceed and instead have an event with an incompatible error, similar to how we check proper csi driver in the csi provisioner. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I think similar issue exists with cloning. We'll need to think about what the behavior should be for data populators too. cc
/triage accepted
Scenario to reproduce the issue Create the nginx deployment and the PVC used by it Exec into the running container, cd into the PVC mounted path and create some files Create the following using as source the The used is the following and wait until it becomes Delete the deployment and the initial PVC Create a new PVC, using the previously created as its Set the name of the new (restored) PVC to the nginx deployment and create the again into the PVC mounted directory. It should contain the previously created files but its empty
can you modify the title of this issue and change to ? In addition to snapshot datasource, using PVC datasource and AnyVolumeDataSource should also fail for in-tree PVC.
/assign
I wasn't 100% sure on data populators. Because it shouldn't have a dependency on CSI right?
I guess even if data populators does support intree volumes, pv controller should ignore it similar to csi provisioner
It may or may not have a dependency on CSI. Design is still in progress. We can leave it out for now until we know exactly how to use it. One thing I know is that the any volume data source cannot be an in-tree k8s resource.
Yes, we can actually just ignore it in pv controller.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm getting same with longhorn, is this issue resolved on your side?


Question: // We need to bind ClusterIP to dummy interface, so set `bindAddr` parameter to `true` in syncService() if err := proxier.syncService(svcNameString, serv, true); err == nil { activeIPVSServices[serv.String()] = true <del> if err := proxier.syncEndpoint(svcName, svcInfo.onlyNodeLocalEndpoints, serv); err != nil { </del> <ins> // ExternalTrafficPolicy only works for NodePort and external LB traffic, does not affect ClusterIP // So we still need clusterIP rules in onlyNodeLocalEndpoints mode. if err := proxier.syncEndpoint(svcName, false, serv); err != nil { </ins> glog.Errorf("Failed to sync endpoint for service: %v, err: %v", serv, err) } } else {

Is this a BUG REPORT or FEATURE REQUEST?: What happened: When setting in ipvs proxy mode, kube-proxy can't create clusterIP ipvs rules. What you expected to happen: Create clusterIP rules, so that can access service from internal. How to reproduce it (as minimally and precisely as possible): Create a service with backends in different node. Set service to . Access the service cluster IP from the node which don't have service backend.


Question: } if _, entryExists := secret.Data[keyName]; entryExists { <del> return fmt.Errorf("cannot add key %s, another key by that name already exists: %v", keyName, secret.Data) </del> <ins> return fmt.Errorf("cannot add key %s, another key by that name already exists", keyName) </ins> } secret.Data[keyName] = data return nil

What happened: What you expected to happen: Human readable error output. Anything else we need to know?: The same is happening when in duplicate key while creating a with binaries. !!! The error output will be all the of the binary itself !!! Environment: Kubernetes version (use ):
/sig cli
/cc


Question: source := &v1.HostPathVolumeSource{ Path: "/tmp", } <del> pod := testPodWithHostVol(volumePath, source, true) </del> <ins> // we can't spawn privileged containers on Windows, nor do we need to. privileged := !framework.NodeOSDistroIs("windows") pod := testPodWithHostVol(volumePath, source, privileged) </ins> // Write the file in the subPath from container 0 container := &pod.Spec.Containers[0]

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --/sig testing /sig windows /area conformance Which jobs are failing: N/A Which test(s) are failing: [sig-storage] HostPath should support r/w [NodeConformance] [sig-storage] HostPath should support subPath [NodeConformance] Since when has it been failing: Before 2019-03-07. Since then, the Windows nodes' kubelet service has been passed the , which allows the test to pass. Testgrid link: (older than 2019-03-07) Reason for failure: The tests are trying to spawn pods with a HostPath volume ( folder has 777 file permissions by default, meaning it should pass without privileged access. The tests have turned green in the linked testgrid because the Windows Kubelet flag ` has been set to true, but it shouldn't have been necessary in the first place. Anything else we need to know:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: e2ewebsocket "k8s.io/kubernetes/test/e2e/framework/websocket" imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" <ins> "k8s.io/utils/pointer" </ins> "github.com/onsi/ginkgo" "github.com/onsi/gomega"

[X] test approval issue : [X] test pr : 28/07/2022 [X] two weeks soak end date : 12/07/2022 [X] test promotion pr : The endpoint as seen on the website is tested but not part of conformance. The APIsnoop query below shows that there is no conformance test for this endpoint. patchCoreV1NamespacedPodStatus Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the pod status endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance /sig node
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale targeting this for 1.25
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: operation := string(sample.Metric["operation_name"]) status := string(sample.Metric["status"]) statusCounts := result.statusMetrics[operation] <ins> metricPluginName := string(sample.Metric["volume_plugin"]) if len(pluginName) > 0 && pluginName != metricPluginName { continue } </ins> switch status { case "success": statusCounts.successCount = count

Which jobs are flaking: () () Which test(s) are flaking: Reason for failure: Timeout: Anything else we need to know: This test flaking about 10% of the time in this job. /sig storage /kind flake /priority important-soon cc
/remove-kind failing-test
/assign
I noticed one issue with the test is that it's not filtering metrics by plugin name. So it will just count the last plugin's metric. I'm not sure if the metrics collected are in any sorted order or not. If not, then when you have a suite that runs multiple plugins, then you might be pulling from the wrong metric.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
still very prevalent in sig-release dashboards
I have pushed which should hopefully fix this.
Also PR against 1.17 branch -


Question: } return nil } <del> // isJournaldAvailable returns whether the system executing the tests uses // journald. func isJournaldAvailable() bool { _, err := exec.LookPath("journalctl") return err == nil } // readinessCheck checks whether services are ready via the supplied health // check URLs. Once there is an error in errCh, the function will stop waiting // and return the error. // TODO(random-liu): Move this to util func readinessCheck(name string, urls []string, errCh <-chan error) error { glog.Infof("Running readiness check for service %q", name) endTime := time.Now().Add(*serverStartTimeout) blockCh := make(chan error) defer close(blockCh) for endTime.After(time.Now()) { select { // We *always* want to run the health check if there is no error on the channel. // With systemd, reads from errCh report nil because cmd.Run() waits // on systemd-run, rather than the service process. systemd-run quickly // exits with status 0, causing the channel to be closed with no error. In // this case, you want to wait for the health check to complete, rather // than returning from readinessCheck as soon as the channel is closed. case err, ok := <-errCh: if ok { // The channel is not closed, this is a real error if err != nil { // If there is an error, return it return err } // If not, keep checking readiness. } else { // The channel is closed, this is only a zero value. // Replace the errCh with blockCh to avoid busy loop, // and keep checking readiness. errCh = blockCh } case <-time.After(time.Second): ready := true for _, url := range urls { resp, err := http.Head(url) if err != nil || resp.StatusCode != http.StatusOK { ready = false break } } if ready { return nil } } } return fmt.Errorf("e2e service %q readiness check timeout %v", name, *serverStartTimeout) } </del>

In node e2e, we have the option to keep the services running after test for further debug. However, we set to for the services, which means that once the parent exits, they will receive and exit with the parent if they don't trap the signal. With this, option will never work. If we still need this feature, we should not set if is false. /cc
We still need this feature. It helps with debugging test failures by keeping all the daemons running upon failure. , Lantao Liu wrote:


Question: } defer newF.Close() f.Close() <del> if err := watcher.Remove(f.Name()); err != nil && !os.IsNotExist(err) { klog.ErrorS(err, "Failed to remove file watch", "path", f.Name()) } </del> f = newF <del> if err := watcher.Add(f.Name()); err != nil { return fmt.Errorf("failed to watch file %q: %v", f.Name(), err) } </del> r = bufio.NewReader(f) } // If the container exited consume data until the next EOF

won't output logs after kubelet rotate log filer of the container. kubectl output logs correctly after the log file is rotated. the config to . So kubelet will rotate the log soon. Build a image with this program a deploy with the previous image. the command No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/area kubelet
/triage accepted /priority important-longterm
/assign since you sent a PR already
Ran into this issue!


Question: // unschedulable, we need to wait until all of them are schedulable. framework.ExpectNoError(framework.WaitForAllNodesSchedulable(c, framework.TestContext.NodeSchedulableTimeout)) <ins> // If NumNodes is not specified then auto-detect how many are scheduleable and not tainted if framework.TestContext.CloudConfig.NumNodes == framework.DefaultNumNodes { framework.TestContext.CloudConfig.NumNodes = len(framework.GetReadySchedulableNodesOrDie(c).Items) } </ins> // Ensure all pods are running and ready before starting tests (otherwise, // cluster infrastructure pods that are being pulled or started can block // test pods from running, and tests that ensure all pods are running and

A) seems to be required by some tests. we need to decide if jobs need to pass explicitly or let the e2e framework detect how many worker nodes a cluster has. the conformance image has no notion of yet. B) the conformance image runs tests even in parallel mode: if "y" is set in the YAML for if we assume that tests should not run in parallel mode, this means we need to pass from the runner. C) should we switch the conformance image to ? currently it's . we need to check if this breaks anything. discussion: /area test /kind design /kind bug /assign
/sig testing
PRs - disable Parallel mode for conformance image (and test):
RE: the introduction of ` in this test broke not only the kubeadm tests but also the conformance image and sonobuoy. previously they did not require and we are now forced to detect the number of nodes outside of the e2e testing framework. possibly this test was the first conformance test that uses as a aggressive failure condition. there are two options here: A) adapt all prow jobs, conformance image and sonobuoy, etc to explicitly pass . this ensures that future test will rely on the number the user provides and expects. we already started going this route by providing mechanics for detecting schedulable nodes, per-solution. B) auto-detect "ready", schedulable nodes in the e2e framework as a start number before, say a daemon set test runs and compare outcomes. this way a test will run on whatever number of nodes are currently available. cc
/area conformance /milestone v1.14
Line should be a explicit failure, not a
i like option B!
related to
Note that this test was NOT new, the other bug is ignoring skipped conformance tests. Ideally that shouldn't be possible.
/assign
GitHub didn't allow me to assign the following users: johnSchnake, stevesloka. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm reluctant to make a sweeping change for 1.14 other then a band aide inside of test-infra. Ideally I'd like to assign this issue to the test-commons group to solve in 1.15. Thoughts?
Agree that (A) (passing num-nodes explicitly) isnt a good option since it breaks other users who dont know about it in the first place and that this is well within the capabilities of the framework. So for option (B) during startup of the framework we need to grab the nodes and count the number that are schedulable and not tainted as master nodes, correct? Save that into a TestContext variable and then reference it elsewhere as needed? Is the plan to reuse the existing value? I'm not at this time familiar with that value and whether it is needed in some other capacity and shouldn't be auto-set. w.r.t. 1.14 vs 1.15; I don't have too strong a preference. I do generally prefer to let things bake a little longer inside of a release but it seems like if just the node count solves the problem its a small, scoped change that could make it in.
From slack: If num-nodes is not specified query on startup and over-ride the test context variable.
a control plane node can be tainted as "master", but still allow workloads. search for on all nodes, and exclude those that have it from the count. this is fed the flag, which is -1 by default. if the user didn't add a value my suggestion is to just update with the detected count. also possibly add notes about that in the flag description. future tests might decide to use this again (we already have some) and it will break if the user doesn't pass a valid number from the CLI flag. the "auto-detect if -1" logic seems reasonable and backwards compatible. how well our detection will work in complicated taint / toleration scenarios is another question. "Code Thaw" starts on the 19th of march. if we merge a change in master before that the change will also be picked by 1.14. if we patch this later the change has to be cherry picked for 1.14.


Question: csiAttacher.watchTimeout = watchTimeout return csiAttacher } <ins> func generateSha(handle string) string { result := sha256.Sum256([]byte(fmt.Sprintf("%s", handle))) return fmt.Sprintf("%x", result) } </ins>

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: package proxy import ( <ins> "strconv" </ins> "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/klog/v2" </ins> ) func TestNodePodCIDRHandlerAdd(t *testing.T) { <ins> oldKlogOsExit := klog.OsExit defer func() { klog.OsExit = oldKlogOsExit }() klog.OsExit = customExit </ins> tests := []struct { name            string oldNodePodCIDRs []string

There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. As suggested by in , we can replace by /sig network /help /kind cleanup
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. /help /kind cleanup </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to raise a PR to fix this.


Question: policies := []batch.ConcurrencyPolicy{batch.AllowConcurrent, batch.ForbidConcurrent, batch.ReplaceConcurrent} *cp = policies[c.Rand.Intn(len(policies))] }, <ins> func(p *batch.PodFailurePolicyOnPodConditionsPattern, c fuzz.Continue) { c.FuzzNoCustom(p) if p.Status == "" { p.Status = api.ConditionTrue } }, </ins> } }

pull-kubernetes-unit: TestRoundTripTypes Unknown No response No response /sig api-machinery
/kind bug /triage accepted
cc
cc I think fuzzing is missing a default for the newly type, so when fuzzing adds an entry, the original doesn't match the round-tripped data (which applies the API default)
/priority important-soon /remove-sig api-machinery /sig scheduling /assign
this will also need picking to release-1.25
/remove-sig scheduling /sig apps
/assign
/unassign


Question: } func (plugin *cniNetworkPlugin) Status() error { <del> // sync network config from confDir periodically to detect network config updates plugin.syncNetworkConfig() </del> // Can't set up pods if we don't have any CNI network configs yet return plugin.checkInitialized() }

What happened: When Node's iops is full, the Node status becomes to be NotReady. The NotReady message is "Container runtime is not ready". After our troubleshoot, we found func in cni package costs more then 30s, then the runtime status health check is failed : What you expected to happen: Node should still be in Ready status when iops is full. Because the CNI config files does not change. How to reproduce it (as minimally and precisely as possible): Run an application to fill disk IO bandwidth, then the Node becomes to be NotReady. Anything else we need to know?: Nothing yet. Environment: Runtime is dockershim. Other environment is no need to pay attention.
/assign /sig-node
/sig node /sig network


Question: claim := templates[i] claim.Name = getPersistentVolumeClaimName(set, &claim, ordinal) claim.Namespace = set.Namespace <del> claim.Labels = set.Spec.Selector.MatchLabels </del> <ins> if claim.Labels != nil { for key, value := range set.Spec.Selector.MatchLabels { claim.Labels[key] = value } } else { claim.Labels = set.Spec.Selector.MatchLabels } </ins> claims[templates[i].Name] = claim } return claims

What happened: When creating a StatefulSet that uses volumeClaimTemplates, the resulting PersistentVolumeClaims have incorrect labels. Instead of using the labels defined in their metadata, they use the labels from the StatefulSet's selector. What you expected to happen: I expect the created PersistentVolumeClaims to have the labels defined in their metadata. How to reproduce it (as minimally and precisely as possible): Resulting PersistentVolumeClaim: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE Install tools: Others: . /sig apps
+1 just ran into this myself.
/assign
GitHub didn't allow me to assign the following users: cwdsuzhou. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
As in . Even thought the StatefulSet controller looks for the PVC by name (not by label) we generate labels for the PVC based on the StatefulSet's selector. Removing this would be a backward incompatible change to a v1 API contract (can't do that). Adding additional labels might be acceptable, but, If you want to change the behavior of the controller wrt to this, I'd like to see a (small) KEP for it so it can be discussed a bit more broadly in the SIG.
I very much think this would be a good discussion to have. If you could, please let me know what's required to start a KEP on this topic. As I see it, the current implementation is in violation of the API: the volumeClaimTemplates data type is listed as " array". A PersistentVolumeClaim's metadata is listed as a standard type. The fact that the PersistentVolumeClaim's metadata is being handled differently when defined in a StatefulSet means that the PersistentVolumeClaim API isn't being properly followed in that context. I'd also like to argue that the change wouldn't be backwards incompatible.

The StatefulSet's volumeClaimTemplates description even says that the matching is done on name: "Every claim in this list must have at least one matching (by name) volumeMount in one container in the template." Respecting the PersistentVolumeClaim's metadata wouldn't necessarily cause any incompatibility from a technical perspective (although it may affect some users' implementations that rely on it). In summary, the current implementation violates the principle of least surprise, makes it impossible to label individual volume claims separately, and it directly contradicts the API specification.
The KEP process is described . You can open a KEP via a PR against the SIG Apps repository. You can use me as your approver. As to the compatibility of the current PR, the current label generation is v1 and has worked the same way for several years now. Users that select the PVCs created by the controller have a reasonable expectation that this should continue to function in subsequent releases. If you add additional labels but generate labels that match the selector this would allow for additional labels while maintaining compatibility (imo).


Question: }) ginkgo.BeforeEach(func() { <del> if err := lookEmitSignalCommand(); err != nil { e2eskipper.Skipf("skipping test because: %v", err) return } </del> ginkgo.By("Wait for the node to be ready") waitForNodeReady() })

xref: Graceful node shutdown test uses to invoke dbus - As per some distros, don't have installed so the test must be skipped in those environments. We should consider if we can switch to use or perhaps and fall back to to ensure the test can work across more distros.
/sig node
/sig testing
/remove-kind bug /kind feature I'm putting kind feature to signify that nothing is really broken in k8s, tests needs improvement.
/remove-sig testing This dbus command should be up to SIG node, though tentatively: /triage accepted
/area test
Do we actually have these other distros available to prevent regressions if we change that?
/assign
the change will be to replace skip logic when is missing to use an alternative way to send this. So there will be no regression on distros where it currently runs without the skip. Original bug should have a link to job that was failing
I would like to work on improving the command. Am I understanding correct here We have to change to logic where we skip to use busctl instead. Could you please give me some idea on how can we make it to use , some similar examples in the repo would be really useful.
please have a look at The issue is that the command available is too old and doesn't include .
I see, it looks like we are not just adding a condition for instead we are replacing it with . It looks you are already working on it. Please inform if I can help in anyway. Thanks for the response.
Both are possible... it was easier to get a CI signal if that was used in all cases. I might put the condition later. I am waiting for comments in the PR, but feel free to propose other solutions that might work with older systemd versions.