Question: return allErrs } <del> func validateIPVSSchedulerMethod(scheduler kubeproxyconfig.IPVSSchedulerMethod, fldPath *field.Path) field.ErrorList { supportedMethod := []kubeproxyconfig.IPVSSchedulerMethod{ kubeproxyconfig.RoundRobin, kubeproxyconfig.WeightedRoundRobin, kubeproxyconfig.LeastConnection, kubeproxyconfig.WeightedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnection, kubeproxyconfig.LocalityBasedLeastConnectionWithReplication, kubeproxyconfig.SourceHashing, kubeproxyconfig.DestinationHashing, kubeproxyconfig.ShortestExpectedDelay, kubeproxyconfig.NeverQueue, "", } allErrs := field.ErrorList{} var found bool for i := range supportedMethod { if scheduler == supportedMethod[i] { found = true break } } // Not found if !found { errMsg := fmt.Sprintf("must be in %v, blank means the default algorithm method (currently rr)", supportedMethod) allErrs = append(allErrs, field.Invalid(fldPath.Child("Scheduler"), string(scheduler), errMsg)) } return allErrs } </del> func validateKubeProxyNodePortAddress(nodePortAddresses []string, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{}

The ipvs mode limit the scheduler algorithm to : now in new kernel have this new algorithm : the are specialy interesting... that be in kernel 4.18...
/sig network /area ipvs
/assign
This seems non-controversial on its face, but a) how do we know whether the kernel supports it or not; and b) if every node makes independent decisions, doesn't this basically become random? /assign uablrek
a) It should be the user's responsibility to ensure that the configured ipvs sheduler is available on all nodes This sounds rude, but please read for some rationale (and ). Exception may be the which is the default, but an installation with but without the sheduler is extremely unlikely. If that relaxing of the sheduler pre-test is acceptable, I would say that any current and future ipvs sheduler shall be allowed. Just treat them as a string. But it should be documented in the release-notes and some place else, but I don't know where.
b) For all shedulers that make decisions based on local state, yes. (that includes btw) Well, not really random ... For instance if is used and a new endpoint is addred, then all ipvs instances will get an "empty" target and will select it for new connections and may overload the newcomer, or it will become "statistically correct" over time. But IMHO users should be allowed to try. Some sheduler may be very useful in their environment. That will (already has) of course spawn some issues when it doesnt work as expected (for the reason in b) for instance), but allowing all shedulers will not change that.
So, my proposal is: Drop the scheduler module check Treat the configured scheduler as a string in the ipvs proxier Document
The module check can be replaced with a function check, e.g. insert/delete an ipvs dummy rule.
people is trying to reduce privileges on containers these days, does it requires more privileges than today or is the same?
The same. Kube-proxy configures ipvs already. This would only add an extra probing insert/delete on start. Even on first start (if ipvs is empty) with some care.
Could even be less privileges if "modprobe" requires any privileges (now or in the future)
What is your use-case?

I have wanted to check the sceduler in ipvs, but havn't got time yet. The load-balancer is stateless, but is not (well, at least not normally). A problem Maglev tries to solve is if the number of Application Servers (AS) (endpoints in K8s) changes, the updated hash buckets must be as few as possible. But if the load-balancer is stateful and stores the connections (like ipvs or the Linux conntrack used in proxy-mode=iptables), no existing connections are affected, no matter what hash algorithm is used for new connections. So, for both proxy-mode ipvs and iptables existing connection are not affected by adding or removing endpoints. But... if external traffic is routed to different nodes, e.g. a GW rehashes ECMP targets after a cluster scaling, then traffic is affected. This is easily tested: Setup a service of type LoadBalancer Route external traffic to the loadBalancerIP to node A Setup a number of TCP connections Re-route external traffic to the loadBalancerIP to node B All connections are lost (I have tested). But if the load-balancer would have been a stateless Maglev no connections would be lost. Google Maglev stores connections but only as a cache and getting the best of stateful, existing connections never lost on AS scaling, and stateless, the load-balancing tier can be scaled with minimum impact. This can be the case in ipvs , but is it?
Lars: proposal SGTM. IPVS impl is pretty much your call to make :) On Thu, Dec 22, 2022, 1:37 AM Lars Ekman
I have started working on a PR to remove the module checks. You are the only active reviewer and I will try my best to divide it into small reviewable commits as you wrote in :smile:
PR to remove the module checks I also intend to remove the constraint on ipvs schedulers, which will fix this issue. If you can create a VS with your configured scheduler you can use it, whatever that scheduler may be.
Looks like it is. Please read I have tested "mh" in kube-proxy (with a patch) and first problem is that is hashes on address only by default, so a traffic generator using the same source all the time will end up on the same endpoint.

This can be configured but will require special treatment in kube-proxy.
Should we close this now that is in?
/triage accepted
No, just removes the module ckeck. The validator still restricts the usable schedulers
to add afinity without using states. is the finality of maglev to avoid perf on managing state... and is more robust on target lost than other consistant hashing method... is normal is what you whant when you use maglev... to have afinity on ip src... but without need of storing states... is afinity by consistant hashing... cilium use this algorithm too....
the google research paper on the algorythm:
If you can build and use an own please try PR I have tested and "mh" can be configured, and with many src-addresses load-balancing works. However if external ingress traffic is re-routed from one K8s node to another, all connections are lost, same as "rr".
with ipvs the node who receive the traffic is considered a director, if you LB to other node... this node must have default route to the director because of nat masquerading of ipvs... when you use MH/RR with Direct Server Return avoid that problem (no more nat masquerading)... the node respond to the client directly (no need to return back to the first node who receive the traffic). but you need to tunnel the traffic from node who receive the request to the node you forward the traffic (cloud flare use fou protocol for that)... i dont know if kubeproxy already support that sort of configuration... cilium for LB DSR + Maglev use tcp option field to store the Virtual IP that receive the traffic in the packet and nat the destination with the pod ip before sending to the node who host the node (that work in full routing mode), and the node respond to the client directly in replacing thaire ip with the Virtual IP from tcp option field.... in that way no state needed... no tunneling... no complex network configuration.... some has been done on kube-router project: and i think you must use in conjonction with MH/RR to use Direct Server Return if i understand whell... and good exemple of the use case : that use ipvs (use ipip tunnel) to do that....
Regarding "mh" algorithm, I submitted PR a few weeks ago which adds the new load distribution method to kube-proxy.

As this commit supports 'mh" with hashing on srcIP and srcPort by default, connections from the same srcIP with different srcPorts will be distributed equally to all EndPoints. If users want to have an affinity on srcIP, they can use "mh" with "Session Affinity" option enabled. The PR is targeted for the main use-cases below: Connection failover from a worker node to another. Scale-in/out worker nodes Rerouting from GWs or external LoadBalancer changes receiving worker nodes of existing traffic flows. I would need to rebase the commit after your fix for this issue PR . PR is now waiting triage state, please verify if it is reasonable to add this new algorithm to kube-proxy.
No, and it probably never will. The proper (only) way to get this into is to write a (KEP). To :
I would also like to see a demo or e2e test working with this setup, it doesn't seem obvious from the comments that this will work out of the box ... The project Kubernetes, as Lars correctly mentions , use KEPs to discuss and document feature, I agree with him and I think this will require to write a KEP
What I meant was your use-case in K8s. Having worked most of 2021 with the Maglev hash algorithm when developing the I know what it can do, but I also know K8s and I think I can say right now; To allow the "mh" in kube-proxy=ipvs will not give you what you hope for.
And a side-remark on DSR. Cilium can do this because they modify packets with eBPF. The traditional way of doing DSR is to actually assign the Virtual IP address (VIP) to an interface in the server (the POD in the K8s case). This must have support in the CNI-plugin, which I guess is the case for as it handles load-balancing and is a CNI-plugin. To add generic support for DSR in K8s is not simple.
I'm new to this community. Thanks for your suggestion about the process of adding a new feature to kube-proxy. I'll prepare a KEP for the "mh" load distribution.
But in any case I would like PR to be merged. For sanity reasons if not others.

The for scheduler=mh can/should be in a separate issue+PR. This will allow users to test "mh" in kube-proxy, because; Nobody will take my word for it :smile:
I don't want to sound discouraging but IMO you should not spend too much time on a KEP before presenting it to get some early feedback. I can't see how "mh" can provide the statelessness unless DSR is used, and that may be very hard to get into the . So, I suggest you describe a use-case in the KEP, and what it takes to support it, and present it on a bi-weekly sig/network meeting. And don't worry, people are nice there :smile:
Perhaps DSR is not needed to use with "mh" since I cannot really see any problem with nat masquerading of ipvs if subsequence packets are sent to the second node. With net/ipv4/vs/sloppy_* enabled, ipvs on the second node accepts the packets as an existing connection and then keeps forwarding them to the original endpoint thanks to the mh algorithm. IPVS of course performs snat and "remember" the client IP of the packets as if they are sent to the first node. When receiving replies from the endpoint, ipvs on the second node will be able to forward the replies to the client with the cached client IP in its NAT table. Please correct me if I'm missing something in this regard.
So even if the kube-proxy forwards packets to the same POD (assuming the same mh configuration in node A and B) the receiving POD will reject with a TCP-RST packet.
Since you have written PR you should be able to test this with you own-built kube-proxy. For me all connections are teminated when re-directing ingress traffic from one K8s node to another.
Thanks for your clarification. Now I could see this problem is on the pod side, not ipvs. It seems that having DSR does not help to solve the problem. Though, I suppose this issue will not happen with the following scenarios: UDP traffic Traffic is sent to Cluster-IP instead of NodeIP
With DSR the src=VIP all the way, so it should be OK, even if masq (DNAT) is used.

UDP same thing, unless your UDP server has no session and doesn't care about addresses, just the server port. This is however not the usual case. Traffic sent to the Cluster-IP will be have src=pod, yes, but it will always be hadled by the local kube-proxy, so the use-case to move traffic to another node doesn't exist for clusterIP.
Me neither :) Agree - you will have the best experience if do this iteratively. Start with an email to the SIG list. Then a doc discussing the peoblem and high-level solution details. THEN a KEP makes sense.
cilium assign vip also but more higher in the kernel stack with ebpf... in that way is more simple to implemente than in ipvs... juste simple packet manipulation (plus the fact that is working only in full routed mode)... ipvs can do that, helped with some config on each node (configuring tunnel for dsr)... for mh usage without dsr we need to add default route on other node to the actual dedicated kubeproxy as l4 director (one node can be designated as director and move to another node when the first one die). is generaly more for loadbalancer/external ip usage....
Is this an acceptable way? merged. This will allow "mh" to be configured as a ipvs scheduler, but only with L3 hashing (no ports). a new issue like "Ipvs scheduler=mh only hashes on addresses, not ports", and implement it in a new (small) PR with FlagSourceHash as have already done in this issue as "done" since "mh" can be configured and as it's implemented in another way. the work towards a KEP to make "mh" work as you want it to work. Including a K8s use-case, DSR (if needed in some form), setting of sysctl's, etc. I leave item 4. to you and . Item 1-3 will enable "mh" but because of NAT, connections are still lost when traffic is re-directed from one node to another (this can be tested without too much effort. I have done it).
Agree with the approach. fixes the raised issue that it enables all distribution methods supported by ipvs in kube-proxy (including "mh" one).

A separate issue for enabling source port hash when using 'mh' sounds reasonable. This is not only applicable for "mh", but also for the "sh" method which kube-proxy is currently supporting as stated in k8s documents. Agree. Sure. I'll need to verify the issue further and check if any easy way possible to avoid the NAT issue on different nodes. IMHO, replacing VIP (virtual IP) with NodeIP seems to be the first option to overcome the issue, but we need to verify its feasibility and impact in k8s.
To allow the "mh" in kube-proxy=ipvs will not give you what you hope for. Nobody will take my word for it :smile: Ok, ok, please let me rephrase: To just allow "mh" in the current ipvs proxier will not give you what you hope for. So, is not enough.
/kind feature


Question: pod := toBeAttachedPod(f) _, err := client.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).To(BeNil()) <ins> err = framework.WaitForPodNameRunningInNamespace(client, pod.Name, f.Namespace.Name) Expect(err).NotTo(HaveOccurred()) </ins> By("'kubectl attach' the pod, should be denied by the webhook") <del> _, err = framework.NewKubectlCommand("attach", fmt.Sprintf("--namespace=%v", f.Namespace.Name), pod.Name, "-i", "-c=container1").Exec() </del> <ins> timer := time.NewTimer(30 * time.Second) defer timer.Stop() _, err = framework.NewKubectlCommand("attach", fmt.Sprintf("--namespace=%v", f.Namespace.Name), pod.Name, "-i", "-c=container1").WithTimeout(timer.C).Exec() </ins> Expect(err).NotTo(BeNil()) if e, a := "attaching to pod 'to-be-attached-pod' is not allowed", err.Error(); !strings.Contains(a, e) { framework.Failf("unexpected 'kubectl attach' error message. expected to contain %q, got %q", e, a)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig testing What happened: The test . This can happen more frequently when it has to pull the necessary image. Additionally, spawning and starting pods takes a bit longer on Windows nodes. â¢ Failure [88.631 seconds] [sig-api-machinery] AdmissionWebhook Should be able to deny attaching pod [It] Oct 8 16:36:05.987: unexpected 'kubectl attach' error message. expected to contain "attaching to pod 'to-be-attached-pod' is not allowed", got "error running &{/usr/local/bin/kubectl [kubectl --server=https://k8s-cbelu- --kubeconfig=/home/ubuntu/kube/config6 attach --namespace=e2e-tests-webhook-d8clc to-be-attached-pod -i -c=container1] [] <nilIf you don't see a command prompt, try pressing enter.nerror: unable to upgrade connection: container container1 not found in pod to-be-attached-pode2e-tests-webhook-d8clcn [] <nil0xc0018941b0 exit status 1 <nil<niltrue [0xc0002d1808 0xc0002d1838 0xc0002d1860] [0xc0002d1808 0xc0002d1838 0xc0002d1860] [0xc0002d1820 0xc0002d1850] [0x92bfb0 0x92bfb0] 0xc00074dd40 <nil}:nCommand stdout:nnstderr:nIf you don't see a command prompt, try pressing enter.nerror: unable to upgrade connection: container container1 not found in pod to-be-attached-pod_e2e-tests-webhook-d8clcnnerror:nexit status 1n" Additionally, the command could hang (if it actually managed to attach), causing the test to never end. The command should be executed with a timeout.

What you expected to happen: The test should wait for the pod to enter a state before trying to attach to it, so it won't randomly fail. The test should have a timeout when trying to attach to the pod. How to reproduce it (as minimally and precisely as possible): Run the test using Windows nodes and / or delete webhook image from the Compute Node's image cache. Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): Windows Server 1803 compute nodes Kernel (e.g. ): N/A Install tools: N/A Others:


Question: --quiet  --delete-disks all  --zone "${ZONE}"  <del> "${MASTER_NAME}" || true </del> <ins> "${MASTER_NAME}" </ins> # Delete the master pd (possibly leaked by kube-up if master create failed) gcloud compute disks delete  --project "${PROJECT}"  --quiet  --zone "${ZONE}"  <del> "${MASTER_NAME}"-pd || true </del> <ins> "${MASTER_NAME}"-pd </ins> # Find out what minions are running. local -a minions

'kubernetes-soak-weekly-deploy' sometimes fails to deploy because the clean-up cluster step fails to delete the template: Then the subsequent cluster step fails to create the template, because it already exists: It keeps retrying and failing, because the template already exists, until it times out and the deployment fails. Suggested fixes: Cluster down should try harder to delete the template (maybe wait/retry). Cluster up should recognize that the template exists, and try to delete/recreate, or just re-use the existing.
Looks like ran in to this on Monday, May 11, 2015 around 1:00 PM PST. I'll assign this to my self and take a crack at fixing it.
This is causing most of our e2e tests to fail in continuous integration as well as users private e2e test clusters. Bumping to priority/p0 and v1.0 milestone. Saad thinks he's figured it out, and is working on a fix as we speak.
In essence... Saad: "I think what's happening is the previous clean up step to delete the managed instance group happens asynchronously, therefore there is a race condition to delete the template. Working on a fix".


Question: github.com/golang/mock v1.6.0 github.com/golangci/golangci-lint v1.51.2 github.com/google/go-flow-levee v0.1.5 <ins> go.uber.org/automaxprocs v1.5.2 </ins> gotest.tools/gotestsum v1.6.4 honnef.co/go/tools v0.4.2 sigs.k8s.io/logtools v0.4.1

<!-- Please only use this template for submitting enhancement requests --What would you like to be : should detect the appropriate GOMAXPROCS based off of the cpu limit supplied to the pod. Why is this needed: If not, test failures can occur (e.g. ) where GOMAXPROCS defaults to the number of cpus on the machine, while the cpu limit is actually far less. Too many compile/test processes launch, and any timing tests can become flaky. The end result is that running the test jobs on a node with more CPUs can make it flaky when running the test on a node with fewer CPUs cleanly passes. It's possible to set GOMAXPROCS=cpu limits in test jobs to avoid this, but it would be nicer if would autodiscover and set GOMAXPROCS appropriately.
/sig testing /sig k8s-infra
/transfer kubernetes
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign dims
xref:
This is already done, which is great, but for some more info I did some benchmarking of setting GOMAXPROCs vs not: Pretty substantially better in some cases


Question: klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s)", nodeResourceGroup, vmName, diskName) ctx, cancel := getContextWithCancel() defer cancel() <del> if _, err := as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM); err != nil { </del> <ins> _, err = as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM) if err != nil { </ins> klog.Errorf("azureDisk - attach disk(%s) failed, err: %v", diskName, err) detail := err.Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) {

What happened: After running a rolling-update on a StatefulSet that uses Azure Disks, part of the replicas remain in a state forever, exhibiting the following events: Restarting the occasionally unblocks the situation: What you expected to happen: Attach/detach operations succeed consistently. How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brAnd the default StorageClass: <details</summary</details</brExtra information: Two other kind of Pod events that are regularly associated with the error above: kubelet logs of a node where attaching the volume is failing kube-controller-manager logs in an environment where attaching the volume is failing Environment: Kubernetes version (use ): v1.10.10 Cloud provider or hardware configuration: Azure (westeurope) OS (e.g. from /etc/os-release): 16.04.4 LTS Kernel (e.g. ): 4.13.0-1018-azure Install tools: ACS-Engine Others: /kind bug /sig storage /sig azure this might be of interest for you, since you were involved in , which was supposed to fix attach/detach issues in Azure.
/assign
fix the attach disk issue in kubelet, while in your case, you restarted the controller manager, and then it works. It looks like disk volume is never attached to vm successfully before you restart controller manager.
Indeed. For completeness I updated the description and 2 more type of events that are regularly associated to this issue.
could you check activity logs in that resource group, the time should be before you restart controller manager, you could check whether there is any disk related issue. Since you have already restarted controller manager, we cannot get all logs before your restart.
I see a bunch of the following failures in the activity log, but the errors stopped 4h before I gathered the logs and restarted the controller-manager: <details<summaryactivity log</summary</details</brI managed to grab the kube-controller-manager logs from an environment where the situation is currently happening. See the updated description.
Hi would you add my slack: andyzhang workspace?

I think your issue is here, you are using strategy According to , if statefulset replica is using azure disk pvc, is not the right way, pls change the strategy to 'Recreate' Update: may be not related to that issue, still in investigation.
In our deployment every Pod uses a distinct Azure Disk, so there is not multi-attachment. I'm still waiting for feedback from customers, hoping this happens again today.
I have tried in my testing env with v1.10.10 on vmss nodes by , there could be error like following after disk attached to that VM, the disk detect action in vmss node seems slow sometimes, while in the end it could be recovered:
thanks for looking into it. In our case the disk never gets attached (1d in ContainerCreating state).
Hi I found an regression issue of , will send a PR soon. Thanks for reporting.


Question: // addTokenSecretReference adds a reference to the ServiceAccountToken that will be created func addTokenSecretReference(refs []v1.ObjectReference) []v1.ObjectReference { <del> return addNamedTokenSecretReference(refs, "default-token-p7w9c") </del> <ins> return addNamedTokenSecretReference(refs, "default-token-stdpg") </ins> } // addNamedTokenSecretReference adds a reference to the named ServiceAccountToken

/kind feature What happened: In playing with the nginx ingress controller deployment, I noticed the following pod name: nginx-ingress-controller--c0ck5 I figure we should possibly put something in place to prevent accidental l33t profanity ;-). What you expected to happen: The following pod name: nginx-ingress-controller--r0ck5
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
My first github (k8s) issue. I've no idea which SIG would cover this topic. /sig product-management
/sig api-machinery
Previously we removed vowels in . We can 3xt3nd the idea with 0, 1 (can stand for ) and 3. would you like to send a PR? If not, I can work on this.
/assign jamesawebb1
Reassign to me if you do not want to contribute a PR.


Question: // original service should work ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (config.clusterIP)", config.EndpointPods[0].Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromEndpointContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromEndpointContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, config.NodeHTTPPort)) <del> config.DialFromEndpointContainer("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromEndpointContainer("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> // Dial second service ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (svc2.clusterIP)", config.EndpointPods[0].Name, svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromEndpointContainer("http", svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromEndpointContainer("http", svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, httpPort)) <del> config.DialFromEndpointContainer("http", config.NodeIP, httpPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromEndpointContainer("http", config.NodeIP, httpPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By("deleting the original node port service") config.DeleteNodePortService() // Second service should continue to function unaffected ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (svc2.clusterIP)", config.EndpointPods[0].Name, svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromEndpointContainer("http", svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromEndpointContainer("http", svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, httpPort)) <del> config.DialFromEndpointContainer("http", config.NodeIP, httpPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromEndpointContainer("http", config.NodeIP, httpPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should update endpoints: http", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> config.DeleteNetProxyPod() ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, config.MaxTries, config.EndpointHostnames()) </del> <ins> err = config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, config.MaxTries, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should update endpoints: udp", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterUDPPort)) <del> config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> config.DeleteNetProxyPod() ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterUDPPort)) <del> config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, config.MaxTries, config.EndpointHostnames()) </del> <ins> err = config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, config.MaxTries, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) // Slow because we confirm that the nodePort doesn't serve traffic, which requires a period of polling.

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: return dsc.updateDaemonSetStatus(ds) } <ins> // hasIntentionalPredicatesReasons checks if any of the given predicate failure reasons // is intentional. func hasIntentionalPredicatesReasons(reasons []algorithm.PredicateFailureReason) bool { for _, r := range reasons { switch reason := r.(type) { case *predicates.PredicateFailureError: switch reason { // intentional case predicates.ErrNodeSelectorNotMatch, predicates.ErrPodNotMatchHostName, predicates.ErrNodeLabelPresenceViolated, // this one is probably intentional since it's a workaround for not having // pod hard anti affinity. predicates.ErrPodNotFitsHostPorts, // DaemonSet is expected to respect taints and tolerations predicates.ErrTaintsTolerationsNotMatch: return true } } } return false } </ins> // nodeShouldRunDaemonPod checks a set of preconditions against a (node,daemonset) and returns a // summary. Returned booleans are: // * wantToRun:

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DaemonSet FailedPlacement Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): OL7.3 Kernel (e.g. ): 4.1.12-94.2.1.el7uek.x86_64 Install tools: Others: What happened: I have a DaemonSet that is only deployed to a subset of nodes in the cluster based on a . I'm seeing events fired with reason for the daemonset pods on nodes that do not fit the selector. What you expected to happen: These events should never fire because they do not meet the node selectors in the pod spec. How to reproduce it (as minimally and precisely as possible): Create a cluster with different sized nodes. E.g. smaller masters than workers. Then deploy the daemonset only to the workers. You will see events triggered for the master nodes that don't meet the requirements. Anything else we need to know:
DaemonSet controller will run predicates to check if node should run daemon pod. In the progress of predicates, it will check if pod fits resources first before other predicates such as PodSelectorMatches. /cc
I think this bug appeared when critical pods were introduced and order of checking predicates changed. The event is useless for daemonsets in current situation :/ I'm not sure if there is a clean way to fix it.
I feel like a clean solution would be to look at the predicate reasons twice. The first time you look at the reasons and keep track if any are not intentional. If all of the reasons are intentional then skip checking the reasons.
actually just changing the order of checks in GeneralPredicates may just work. I'm testing the fix now.
Ok, GeneralPredicates returns all reasons. Error that pod doesn't match the node is also returned but because it's "intentional" there is no event for it.

The event "Node didn't have enough resource" should be sent only when there are no other "intentional" reasons.
Yes, I think maybe we need to check reason list twice: check is in tail of predicate list.
/cc
any PR for this? If not, I can help
I will have some time to work on it next week. If you have a spare time you can create a PR.
/assign


Question: --port="$KUBELET_PORT"  --healthz_port="${KUBELET_HEALTHZ_PORT}" 1>&2 & KUBELET_PID=$! <del> kube::util::wait_for_url "http://127.0.0.1:${KUBELET_HEALTHZ_PORT}/healthz" "kubelet: " 0.2 25 </del> <ins> kube::util::wait_for_url "http://127.0.0.1:${KUBELET_HEALTHZ_PORT}/healthz" "kubelet(masterless)" </ins> kill ${KUBELET_PID} 1>&2 2>/dev/null kube::log::status "Starting kubelet in masterful mode"

Example:
Is kubelet healthz coupled with the docker daemon being accessible? ?
That particular error is coming from cAdvisor and should not be causing the test to block. The Kubelet does wait up to 5min for Docker to come up though. Do we use a real Docker in these tests?
Shippable is running on different os dist? ContainerOsVersion:Ubuntu precise?
Yes today kubelet healthz coupled with docker daemon. We even periodically healthcheck docker and kubelet, and restart them even there are issues.
Another example:
bumping priority because we have to get these flaky tests fixed.
This has been marked P0 for 2 months but hasn't had any updates. Is this still happening on Shippable? If so, can we just increase a timeout in the test?
This is ridiculous, I'll fix it myself.


Question: return result } <del> func (t *timeBudget) periodicallyRefresh(stopCh <-chan struct{}) { </del> <ins> func (t *timeBudgetImpl) periodicallyRefresh(stopCh <-chan struct{}) { </ins> ticker := time.NewTicker(time.Second) defer ticker.Stop() for {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: } } <ins> func TestRunAttach(t *testing.T) { tests := []struct { name                string rm                  bool quiet               bool deleteErrorMessage  string expectedDeleteCount int expectedOut         string expectedErrOut      string }{ { name:                "test attach", rm:                  false, quiet:               false, expectedDeleteCount: 0, expectedOut:         "", expectedErrOut:      "If you don't see a command prompt, try pressing enter.n", }, { name:                "test attach with quiet", rm:                  false, quiet:               true, expectedDeleteCount: 0, expectedOut:         "", expectedErrOut:      "", }, { name:                "test attach with rm", rm:                  true, quiet:               false, expectedDeleteCount: 1, expectedOut:         "pod "foo" deletedn", expectedErrOut:      "If you don't see a command prompt, try pressing enter.n", }, { name:                "test attach with rm should not print message if quiet is specified", rm:                  true, quiet:               true, expectedDeleteCount: 1, expectedOut:         "", expectedErrOut:      "", }, { name:                "error should be displayed if delete fails", rm:                  true, quiet:               false, deleteErrorMessage:  "delete error message", expectedDeleteCount: 1, expectedOut:         "", expectedErrOut:      "If you don't see a command prompt, try pressing enter.nDelete failed: delete error messagen", }, } fakePod := &corev1.Pod{ TypeMeta: metav1.TypeMeta{ APIVersion: "v1", Kind:       "Pod", }, ObjectMeta: metav1.ObjectMeta{ Name:      "foo", Namespace: "default", }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "bar", }, }, }, Status: corev1.PodStatus{ Phase: corev1.PodRunning, Conditions: []corev1.PodCondition{ { Type:   corev1.PodReady, Status: corev1.ConditionTrue, }, }, }, } for _, test := range tests { t.Run(test.name, func(tt *testing.T) { postCount := 0 attachCount := 0 deleteCount := 0 attachFunc = func(o *attach.AttachOptions, containerToAttach *corev1.Container, raw bool, sizeQueue remotecommand.TerminalSizeQueue) func() error { if containerToAttach.Name != "bar" { tt.Fatalf("expected attach to container name "bar", but got %q", containerToAttach.Name) } return func() error { attachCount++ return nil } } tf := cmdtesting.NewTestFactory().WithNamespace("test") defer tf.Cleanup() codec := scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...) ns := scheme.Codecs.WithoutConversion() tf.Client = &fake.RESTClient{ GroupVersion:         schema.GroupVersion{Version: ""}, NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { switch p, m := req.URL.Path, req.Method; { case m == "POST" && p == "/namespaces/test/pods": postCount++ body := cmdtesting.ObjBody(codec, fakePod) return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: body}, nil case m == "GET" && p == "/api/v1/namespaces/default/pods": event := &metav1.WatchEvent{ Type:   "ADDED", Object: runtime.RawExtension{Object: fakePod}, } body := cmdtesting.ObjBody(codec, event) return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: body}, nil case m == "GET" && p == "/namespaces/default/pods/foo": body := cmdtesting.ObjBody(codec, fakePod) return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: body}, nil case m == "DELETE" && p == "/namespaces/default/pods/foo": deleteCount++ if test.deleteErrorMessage != "" { body := cmdtesting.ObjBody(codec, &metav1.Status{ Status:  metav1.StatusFailure, Message: test.deleteErrorMessage, }) return &http.Response{StatusCode: http.StatusInternalServerError, Header: cmdtesting.DefaultHeader(), Body: body}, nil } else { body := cmdtesting.ObjBody(codec, fakePod) return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: body}, nil } default: tt.Errorf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, fmt.Errorf("unexpected request") } }), } tf.ClientConfigVal = &restclient.Config{ ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Version: ""}, NegotiatedSerializer: ns}, } streams, _, bufOut, bufErr := genericclioptions.NewTestIOStreams() cmdutil.BehaviorOnFatal(func(str string, code int) { bufErr.Write([]byte(str)) }) cmd := NewCmdRun(tf, streams) cmd.Flags().Set("image", "test-image") cmd.Flags().Set("attach", "true") if test.rm { cmd.Flags().Set("rm", "true") } if test.quiet { cmd.Flags().Set("quiet", "true") } parentCmd := cobra.Command{} parentCmd.AddCommand(cmd) cmd.Run(cmd, []string{"test-pod"}) if postCount != 1 { tt.Fatalf("expected 1 post request, but got %d", postCount) } if attachCount != 1 { tt.Fatalf("expected 1 attach call, but got %d", attachCount) } if deleteCount != test.expectedDeleteCount { tt.Fatalf("expected %d delete requests, but got %d", test.expectedDeleteCount, deleteCount) } if bufErr.String() != test.expectedErrOut { tt.Fatalf("unexpected error. got: %q, expected: %q", bufErr.String(), test.expectedErrOut) } if bufOut.String() != test.expectedOut { tt.Fatalf("unexpected output. got: %q, expected: %q", bufOut.String(), test.expectedOut) } }) } } </ins> func TestRunOverride(t *testing.T) { tests := []struct { name           string

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: LivenessProbe  *Probe               `json:"livenessProbe,omitempty"` ReadinessProbe *Probe               `json:"readinessProbe,omitempty"` Lifecycle      *Lifecycle           `json:"lifecycle,omitempty"` <del> // Optional: Defaults to /dev/termination-log </del> <ins> // Required. </ins> TerminationMessagePath string `json:"terminationMessagePath,omitempty"` // Optional: Default to false. Privileged bool `json:"privileged,omitempty"` <del> // Optional: Policy for pulling images for this container </del> <ins> // Required: Policy for pulling images for this container </ins> ImagePullPolicy PullPolicy `json:"imagePullPolicy"` // Optional: Capabilities for container. Capabilities Capabilities `json:"capabilities,omitempty"`

Many comments in there say "if not specified, default is ..." - but we previously asserted that there are no defaults for the internal API. We should strip those comments out, and leave them in versioned structs only.
cc


Question: return waitForNodeToBe(c, name, false, timeout) } <ins> func isNodeReadySetAsExpected(node *api.Node, wantReady bool) bool { // Check the node readiness condition (logging all). for i, cond := range node.Status.Conditions { Logf("Node %s condition %d/%d: type: %v, status: %v", node.Name, i+1, len(node.Status.Conditions), cond.Type, cond.Status) // Ensure that the condition type is readiness and the status // matches as desired. if cond.Type == api.NodeReady && (cond.Status == api.ConditionTrue) == wantReady { Logf("Successfully found node %s readiness to be %t", node.Name, wantReady) return true } } return false } </ins> // waitForNodeToBe returns whether node name's readiness state matches wantReady // within timeout. If wantReady is true, it will ensure the node is ready; if // it's false, it ensures the node is in any state other than ready (e.g. not

I ran e2e tests against docker 1.7.0-rc2 release, found a bunch of issues. Here is one I think it belongs to kubernetes for sure. 1) ResizeNode test failed, the machine is removed from instance group: But kubectl get nodes still returns 3 ndoes: This might be caused by the new docker binary I am testing against here. This is not the issue I am filing here. 2) Now networking test is running later, I noticed that the test is putting a pod on each node: nettest-gwtc4 on node e2e-test-dawnchen-minion-5l11 I haven't checked networking code in detail yet, not sure the issue is test itself, or scheduler issue. cc/ since you recently changed this. cc/
The networking test doesn't use the scheduler, it manually fills in the host field.
Ok, then we should filter out NotReady nodes from testing; otherwise, the test itself has the dependencies on other tests.
Yes, that test is doing a questionable thing.
Agreed.
I implemented skipping not-ready nodes in . Is it enough to close this issue, or is there something else to be done here?
I think that should fix the issue. ?
status: waiting for merge of


Question: } return scheme } <del> errExpectedFn := func(v int32, msg string) field.ErrorList { return field.ErrorList{ field.Invalid(field.NewPath("spec").Child("limited").Child("nominalConcurrencyShares"), int32(v), msg), } } </del> tests := []struct { <del> name               string obj                runtime.Object old                *flowcontrol.PriorityLevelConfiguration // for UPDATE only zeroFeatureEnabled bool scheme             *runtime.Scheme errExpected        field.ErrorList </del> <ins> name        string obj         runtime.Object old         *flowcontrol.PriorityLevelConfiguration // for UPDATE only scheme      *runtime.Scheme errExpected field.ErrorList </ins> }{ { <del> name:               "v1, feature disabled, create, zero value, error expected", obj:                v1ObjFn(ptr.To(int32(0))), zeroFeatureEnabled: false, scheme:             v1SchemeFn(t), errExpected:        errExpectedFn(0, "must be positive"), }, { name:               "v1, feature disabled, create, unset, no error expected", obj:                v1ObjFn(nil), zeroFeatureEnabled: false, scheme:             v1SchemeFn(t), errExpected:        nil, }, { name:               "v1, feature disabled, create, non-zero, no error expected", obj:                v1ObjFn(ptr.To(int32(1))), zeroFeatureEnabled: false, scheme:             v1SchemeFn(t), errExpected:        nil, }, { name:               "v1, feature enabled, create, zero value, no error expected", obj:                v1ObjFn(ptr.To(int32(0))), zeroFeatureEnabled: true, scheme:             v1SchemeFn(t), errExpected:        nil, }, { name:               "v1, feature enabled, create, unset, no error expected", obj:                v1ObjFn(nil), zeroFeatureEnabled: true, scheme:             v1SchemeFn(t), errExpected:        nil, }, { name:               "v1, feature enabled, create, non-zero, no error expected", obj:                v1ObjFn(ptr.To(int32(1))), zeroFeatureEnabled: true, scheme:             v1SchemeFn(t), errExpected:        nil, }, { name:               "v1beta3, feature disabled, create, zero value, error expected", obj:                v1beta3ObjFn(0, true), zeroFeatureEnabled: false, scheme:             v1beta3SchemeFn(t), errExpected:        errExpectedFn(0, "must be positive"), </del> <ins> name:        "v1, create, zero value, no error expected", obj:         v1ObjFn(ptr.To(int32(0))), scheme:      v1SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1beta3, feature disabled, create, zero value without annotation, no error expected", obj:                v1beta3ObjFn(0, false), zeroFeatureEnabled: false, scheme:             v1beta3SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1, create, unset, no error expected", obj:         v1ObjFn(nil), scheme:      v1SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1beta3, feature disabled, create, non-zero, no error expected", obj:                v1beta3ObjFn(1, false), zeroFeatureEnabled: false, scheme:             v1beta3SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1, create, non-zero, no error expected", obj:         v1ObjFn(ptr.To(int32(1))), scheme:      v1SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1beta3, feature enabled, create, zero value, no error expected", obj:                v1beta3ObjFn(0, true), zeroFeatureEnabled: true, scheme:             v1beta3SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1beta3, create, zero value, no error expected", obj:         v1beta3ObjFn(0, true), scheme:      v1beta3SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1beta3, feature enabled, create, zero value without annotation, no error expected", obj:                v1beta3ObjFn(0, false), zeroFeatureEnabled: true, scheme:             v1beta3SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1beta3, create, zero value without annotation, no error expected", obj:         v1beta3ObjFn(0, false), scheme:      v1beta3SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1beta3, feature enabled, create, non-zero, no error expected", obj:                v1beta3ObjFn(1, false), zeroFeatureEnabled: true, scheme:             v1beta3SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1beta3, create, non-zero, no error expected", obj:         v1beta3ObjFn(1, false), scheme:      v1beta3SchemeFn(t), errExpected: nil, </ins> }, // the following use cases cover UPDATE { <del> name:               "v1, feature disabled, update, zero value, existing has non-zero, error expected", obj:                v1ObjFn(ptr.To(int32(0))), old:                internalObjFn(1), zeroFeatureEnabled: false, scheme:             v1SchemeFn(t), errExpected:        errExpectedFn(0, "must be positive"), </del> <ins> name:        "v1, update, zero value, existing has non-zero, no error expected", obj:         v1ObjFn(ptr.To(int32(0))), old:         internalObjFn(1), scheme:      v1SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1, feature disabled, update, zero value, existing has zero, no error expected", obj:                v1ObjFn(ptr.To(int32(0))), old:                internalObjFn(0), zeroFeatureEnabled: false, scheme:             v1SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1, update, zero value, existing has zero, no error expected", obj:         v1ObjFn(ptr.To(int32(0))), old:         internalObjFn(0), scheme:      v1SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1, feature disabled, update, non-zero value, existing has zero, no error expected", obj:                v1ObjFn(ptr.To(int32(1))), old:                internalObjFn(0), zeroFeatureEnabled: false, scheme:             v1SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1, update, non-zero value, existing has zero, no error expected", obj:         v1ObjFn(ptr.To(int32(1))), old:         internalObjFn(0), scheme:      v1SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1, feature enabled, update, zero value, existing has non-zero, no error expected", obj:                v1ObjFn(ptr.To(int32(0))), old:                internalObjFn(1), zeroFeatureEnabled: true, scheme:             v1SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1beta3, update, zero value, existing has non-zero, no error expected", obj:         v1beta3ObjFn(0, true), old:         internalObjFn(1), scheme:      v1beta3SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1, feature enabled, update, zero value, existing has zero, no error expected", obj:                v1ObjFn(ptr.To(int32(0))), old:                internalObjFn(0), zeroFeatureEnabled: true, scheme:             v1SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1beta3, update, zero value, existing has zero, no error expected", obj:         v1beta3ObjFn(0, true), old:         internalObjFn(0), scheme:      v1beta3SchemeFn(t), errExpected: nil, </ins> }, { <del> name:               "v1, feature enabled, update, non-zero value, existing has zero, no error expected", obj:                v1ObjFn(ptr.To(int32(1))), old:                internalObjFn(0), zeroFeatureEnabled: true, scheme:             v1SchemeFn(t), errExpected:        nil, }, { name:               "v1beta3, feature disabled, update, zero value, existing has non-zero, error expected", obj:                v1beta3ObjFn(0, true), old:                internalObjFn(1), zeroFeatureEnabled: false, scheme:             v1beta3SchemeFn(t), errExpected:        errExpectedFn(0, "must be positive"), }, { name:               "v1beta3, feature disabled, update, zero value, existing has zero, no error expected", obj:                v1beta3ObjFn(0, true), old:                internalObjFn(0), zeroFeatureEnabled: false, scheme:             v1beta3SchemeFn(t), errExpected:        nil, }, { name:               "v1beta3, feature disabled, update, non-zero value, existing has zero, no error expected", obj:                v1beta3ObjFn(1, false), old:                internalObjFn(0), zeroFeatureEnabled: false, scheme:             v1beta3SchemeFn(t), errExpected:        nil, }, { name:               "v1beta3, feature enabled, update, zero value, existing has non-zero, no error expected", obj:                v1beta3ObjFn(0, true), old:                internalObjFn(1), zeroFeatureEnabled: true, scheme:             v1beta3SchemeFn(t), errExpected:        nil, }, { name:               "v1beta3, feature enabled, update, zero value, existing has zero, no error expected", obj:                v1beta3ObjFn(0, true), old:                internalObjFn(0), zeroFeatureEnabled: true, scheme:             v1beta3SchemeFn(t), errExpected:        nil, }, { name:               "v1beta3, feature enabled, update, non-zero value, existing has zero, no error expected", obj:                v1beta3ObjFn(1, false), old:                internalObjFn(0), zeroFeatureEnabled: true, scheme:             v1beta3SchemeFn(t), errExpected:        nil, </del> <ins> name:        "v1beta3, update, non-zero value, existing has zero, no error expected", obj:         v1beta3ObjFn(1, false), old:         internalObjFn(0), scheme:      v1beta3SchemeFn(t), errExpected: nil, </ins> }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { <del> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ZeroLimitedNominalConcurrencyShares, test.zeroFeatureEnabled)() </del> scheme := test.scheme scheme.Default(test.obj)

In 1.30, Priority and Fairness should allow a zero value, either via v1 or v1beta3 (with the roundtrip annotation) in create and update requests See:
/sig api-machinery /cc /assign
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accpted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: require ( github.com/coreos/etcd v3.3.15+incompatible github.com/emicklei/go-restful v2.9.5+incompatible <ins> github.com/ghodss/yaml v0.0.0-20180820084758-c7ce16629ff4 // indirect </ins> github.com/go-openapi/errors v0.19.2 github.com/go-openapi/spec v0.19.2 github.com/go-openapi/strfmt v0.19.0

What happened: Created a CRD with a group/version/kind that defines a CustomResourceDefinition type overlapping with the in-tree kubernetes type (WARNING: do not create this CRD on a cluster you care about, see ) Note that two definitions for the group/version/kind appear in openapi: What you expected to happen: published openapi only includes the in-tree type, which has precedence /cc /area custom-resources /priority important-soon /sig api-machinery
/cc
haven't looked into the code yet. I'd expect the naming controller / apiextensions server to prevent conflicting CR GVK, but they are unaware of builtin API groups
CRD handling is isolated to the apiextensions server... it does not have information about all other built-in and aggregated APIs
mentioned something he was working on to make the aggregation layer honor APIService priorities (so that later definitions would not be included if something had already defined a GVK) edit: possibly
a naive idea is to reserve k8s groups in a hard code way that sounds good the CRD in defines a but the kubectl results are about . Did I miss something?
oops... fixed. I started with an API type under but the issue only surfaces for specific APIs (due to the generated definition name colliding for some types)
that wouldn't address duplicate GVKs from aggregated APIs
Three kinds of openapi paths/definitions that we want to prevent CRDs from overwriting: CRD API the spec aggregation in apiextensions-apiserver allows overwriting. I will send a fix native APIs well protected. The spec aggregation in kube-aggregator: 1) ignores duplicated paths and renames duplicated definitions; 2) treats kube-apiserver with than apiextensions-apiserver APIs the spec aggregation in kube-aggregator allows overwriting. The spec aggregation in kube-aggregator treats aggregated APIs with than apiextensions-apiserver. I'm thinking about giving apiextensions-apiserver the lowest priority, but then the problem is aggregated APIs can potentially overwrite the CRD API Note that none of the above prevents CRDs from adding new subpaths to existing APIs. E.g. a CRD can add a path to the CRD API, or to a aggregated API if it didn't have one


Question: e2eskipper.Skipf("Could not get controller-manager metrics - skipping") } <del> // Forced detach metric should be present forceDetachKey := "attachdetach_controller_forced_detaches" _, ok := updatedControllerMetrics[forceDetachKey] if !ok { framework.Failf("Key %q not found in A/D Controller metrics", forceDetachKey) } </del> // Wait and validate totalVolumesKey := "attachdetach_controller_total_volumes" states := []string{"actual_state_of_world", "desired_state_of_world"}

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
3 tests seem to be failing related to sig-storage: <img width="1229" alt="image" src=""
Also see there are a few there as well
/assign
x-ref Looks like the only jobs running volume metrics tests are e2e-ci-kubernetes-e2e-cos-gce-serial-canary &e2e-ci-kubernetes-e2e-al2023-aws-serial-canary... In that PR we are trying to make those tests run in for kind clusters and gce-serial. Therefore [sig-storage] [Serial] Volume metrics PVC should create metrics for total number of volumes in A/D Controller has likely been broken for a while. Is the metric for being emitted in that test at all? I printed the metricsGrabber results while running test on a kOps cluster and only saw . Could have printed wrong though*
Thanks for dumping the metrics, good call. Could we be dealing with a logical error in the test? I would expect that the only requirement for passing a test to be that the metric is emitted (which it is, as per the dump) It's odd to also expect in . Ideally, this validation: should be part of a separate test.
cc
Perhaps later an extra serial test where force detach is actually excercised should be . But I think the short-term question we should be trying to solve next is why the forced detach metric is not present in this test, and if that is a regression vs this test not functioning correctly for last few years since line was . ~If I were consuming these metrics, I would expect that there is an metric emitted, even if the value is 0, if any volumes were attached/detached for a particular driver. Right? (But Perhaps metric only emitted lazily on first detach?

Will investigate deeper)~ Never mind, it makes sense for this metric to be missing if force detach never happened, and perhaps this test has just been broken for 6 years since this line was . I'll go put up a PR.


Question: "github.com/aws/aws-sdk-go/service/kms" "github.com/golang/glog" "github.com/prometheus/client_golang/prometheus" <ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/record" </ins> "path" "k8s.io/api/core/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/client-go/kubernetes/scheme" v1core "k8s.io/client-go/kubernetes/typed/core/v1" </ins> "k8s.io/kubernetes/pkg/api/v1/service" "k8s.io/kubernetes/pkg/cloudprovider" "k8s.io/kubernetes/pkg/controller"

On AWS when we call "attach" of a volume then we periodically we check (via API call) if volume has indeed been attached. We run the check in a loop with timeout of 21 minutes. AWS documents that - if a volume is stuck in "attaching" state for too long, the node where volume is being attached is in "dirty" state and must be rebooted to fix the problem. Until node is rebooted, more pods with volumes to same node is only going be stuck in same state (i.e "attaching"). We propose that - if attach operation times out waiting for volume in "attaching" state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node. /sig aws /sig storage cc


Question: forbidden("spec", "validation", "openAPIV3Schema", "properties[bar]", "items", "nullable"), }, }, <ins> { name: "suppress per-expression cost limit in pre-existing versions", old: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group: "group.com", Scope: apiextensions.ResourceScope("Cluster"), Names: apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Versions: []apiextensions.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, { Name: "v2", Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, resource: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group: "group.com", Scope: apiextensions.ResourceScope("Cluster"), Names: apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Versions: []apiextensions.CustomResourceDefinitionVersion{ { Name:    "v1", // unchanged Served:  true, Storage: true, Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, { Name: "v2", // touched Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7] + self[8]`, }, }, }, }, }, }, }, { Name: "v3", // new Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, errors: []validationMatch{ // versions[0] is exempted because it existed in oldObject forbidden("spec", "versions[1]", "schema", "openAPIV3Schema", "properties[f]", "x-kubernetes-validations[0]", "messageExpression"), forbidden("spec", "versions[2]", "schema", "openAPIV3Schema", "properties[f]", "x-kubernetes-validations[0]", "messageExpression"), }, }, { name: "suppress per-expression cost limit in new object during top-level schema to Versions extraction", old: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group:   "group.com", Scope:   apiextensions.ResourceScope("Cluster"), Names:   apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Version: "v1", Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, resource: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "plural.group.com", ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Group: "group.com", Scope: apiextensions.ResourceScope("Cluster"), Names: apiextensions.CustomResourceDefinitionNames{Plural: "plural", Singular: "singular", Kind: "Plural", ListKind: "PluralList"}, Versions: []apiextensions.CustomResourceDefinitionVersion{ { Name:    "v1", // unchanged, was top-level Served:  true, Storage: true, Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7]`, }, }, }, }, }, }, }, { Name: "v2", // new Schema: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "f": { Type: "array", Items: &apiextensions.JSONSchemaPropsOrArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "string", }, }, XValidations: apiextensions.ValidationRules{ { Rule:              "true", MessageExpression: `self[0] + self[1] + self[2] + self[3] + self[4] + self[5] + self[6] + self[7] + self[8]`, }, }, }, }, }, }, }, }, }, Status: apiextensions.CustomResourceDefinitionStatus{StoredVersions: []string{"v1"}}, }, errors: []validationMatch{ // versions[0] is exempted because it existed in oldObject as top-level schema. forbidden("spec", "versions[1]", "schema", "openAPIV3Schema", "properties[f]", "x-kubernetes-validations[0]", "messageExpression"), }, }, </ins> } for _, tc := range tests {

We were using cel-go v0.12.7 in Kubernetes 1.27 release and cel-go v0.16.0 in Kubernetes 1.28 release. There are many changes from cel-go side between those two releases and some may break backward compatible of CRD validation rules. An example would be the estimated cost for CRD validation rules(Thanks to who brought this to our attention with the following example). In the example CRD validation rules below: It will pass perfect in Kubernetes 1.27 version but fail in Kubernetes 1.28 with error: This is due to the estimator fix happened in cel-go(, ) The changes should not cause backward incompatible of CRD validation rules. OR a stability test suite should be in place to prevent the changes causing backward compatible issue before GA. Apply the above CRD in Kubernetes 1.27 and 1.28, 1.27 cluster should pass and 1.28 cluster should fail. Since the fix in cel-go side makes sense to me and the feature is in Beta now, I am not quite sure how strict we should be on backward compatible. cc As a followup, a stability test suite for estimate cost is worth adding before GA to make sure any future changes be caught before the feature promoted to GA. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I agree that the changes in cel-go should not break the backward compatibility of CRD validation rules, especially for users who have already defined and deployed their custom resources using the previous version of Kubernetes. One suggestion I have is to use a version flag for the CEL library, so that users can specify which version of cel-go they want to use for their CRD validation rules. This way, they can avoid unexpected errors or behavior changes when upgrading their Kubernetes cluster.

For example, they can add something like in their CRD schema to indicate that they want to use the old version of cel-go.
Another suggestion I have is to use a cost limit parameter for the CEL library, so that users can adjust the budget for their CRD validation rules. This way, they can fine-tune the trade-off between performance and complexity of their rules. For example, they can add something like in their CRD schema to indicate that they want to allow more expensive rules.
I agree this is the type of bug fix that is legitimate to make during beta. when the size changes came in, did our existing size estimation test coverage notice this cost increased? If not, is there a systemic change we could make to exercise each type of estimation combination so we have visibility to any future changes. Existing persisted CRD objects should continue to work as-is on upgrade, but the CEL rules and/or schema would be required to fit within fixed cost estimates before making future updates/changes to the CEL rules there should be unit tests ensuring updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates, but can you verify that is the case? That's not workable, both from an implementation perspective and from a fragmentation perspective. We explicitly don't want to allow opting into loopholes / bugs in cost estimation that allow CRD authors to make API servers do excessive amounts of work. Similar to above, the responsibility for API server resource consumption lies with the kube-apiserver admin, not the CRD author, so exposing this in the API is not something I would expect.
/assign
Thank you! I guess the next step would be: the existing test suite regarding with the cel cost evaluation. Add estimated cost tests for every single operation, function, type, etc if it's not existing now. updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates. Currently it is not working like this but would definitely help here. cc , please correct me if I'm wrong :) Those should be done before the feature goes to GA. I am not sure about the policy of a GA feature relying on alpha/beta feature is acceptable or not. is going to Beta in this release. The worst case is we have to wait further until the feature is ready.

I don't think that helps... CRD ratcheting is about validation of the custom resources. This is about validation of the CRD itself.
thanks for clarifying! Are you suggesting while updating CRD, if the rules matching what's in the existing object, we should skip cost estimation check? That makes sense. I could start a PR if my understanding aligns :)
something along those lines, though now that I think about it, other changes in the schema could affect the number of times a CEL expression could be run (dropping or increasing maxItems, maxProperties, etc) we already collect preexisting expressions when updating a CRD in order to allow already-persisted expressions (possibly persisted via a newer API server) making use of new CEL features: I think that is what I was thinking of. I would probably focus on: coverage of the edges of the cost estimation so we don't pick up changes there without knowing coarse-grained skipping of schema/cel validation for a particular version in a CRD for updates which don't change anything under that version's schema (would let someone make unrelated updates to the CRD and leave an existing version's schema / CEL rules alone)
Thank you! We have those work planned for the current cycle. /assign /assign


Question: const UserNamespacesSupportKernelVersion = "6.3" const TmpfsNoswapSupportKernelVersion = "6.4" <ins> // NFTablesKubeProxyKernelVersion is the lowest kernel version kube-proxy supports using // nftables mode with by default. This is not directly related to any specific kernel // commit; see https://issues.k8s.io/122743#issuecomment-1893922424 const NFTablesKubeProxyKernelVersion = "5.13" </ins>

continuing discussion from and
/sig network /triage accepted
from I think we'll officially declare that we support kernel 5.4+ and nftables command-line 0.9.2+, and we can use a expression to check that. (Which means really we'll support kernel 5.3, but we won't advertise it that way.) FTR (and maybe worth documenting in a comment wherever we put the check): If we decide we need concatenations-with-ranges support again later, that would imply 5.6/0.9.4. If we want to use in sets/maps, that would require at least 0.9.4 (and maybe later depending on exactly how we're using it; we'd have to re-check the release notes). Support for comments on tables/chains/sets/maps/etc (anything but rules and elements) didn't come until 5.10/0.9.7, but knftables itself already does a for that at construct time, and just drops those comments if they're not supported.
There was a and about nft segfaults. It appears that specifically, nft v0.9.4 to v0.9.8 (inclusive) will segfault if asked to do anything when any chain in any table contains a certain kind of set/map as created by nft v1.0.3 or later. Specifically, rules like these trigger it (I think specifically it's caused by using and together, which the anonymous map as-generated-by-1.0.3+ uses. Using a named map rather than an anonymous one does not help.) Note that with nft < 1.0.1, doing any operation requires parsing the entire current ruleset. So that means when kube-proxy is using nft 1.0.3 or newer, nothing else in the host network namespace can use nft 0.9.4 - 0.9.8 for anything_. if the system firewall or something else in the host network namespace is using nft 1.0.3 or newer, then it might create rules that will break kube-proxy if kube-proxy is using nft 0.9.4 - 0.9.8.

If kube-proxy is using nft 1.0.1 or later, then it should not be possible for any other well-behaved component to break it, even if there are future bugs like the current one, because kube-proxy never looks at any tables but its own, so it wouldn't see things that other components did that it couldn't understand. OK, so what does this mean for us? We could just ship the official kube-proxy images with hand-built nft 0.9.2 rather than whatever's current in Debian. Then we guarantee we don't install rules other people can't deal with. Except that then we potentially run into problems if the system uses a much newer nft with some new bug that breaks 0.9.2. (We could do something -esque here, but ... ugh.) I think we actually want to go the opposite way, and have kube-proxy print a warning at startup if your nft is older then 1.0.1, because older versions run the risk of being broken by something someone else is doing in another table. But then there's the problem of us breaking other people. Or, even if we don't break other software, we still make it that admins on some systems would be unable to debug nftables-based problems... I guess we could avoid using the known-problematic syntax... the simplest way to do this would be to revert back to doing endpoints the way that iptables does: of course, that's less efficient, but...
Sounds to me that 1.0.2 is the golden version. It doesn't break 0.9.8, and it doesn't parse the entire current ruleset.
golden relative to the bugs we know about right now anyway... but maybe building our official images with 1.0.2 for a while would be a good plan... (Unfortunately we would have to actually build it ourselves. We can't just pull a 1.0.2 build from a random older Debian release because Debian never shipped that version; bullseye had 0.9.8 and bookworm had 1.0.6.)
thoughts on "use less efficient rules" vs "build images with a very specific nft version"?

build images with nft 1.0.1 seems the best option to me , because it seems that nft < 1.0.1 is not prod ready based on the different behaviors we are observing. If this happens between minors in 1.x.y then I think is a different discussion
The question wasn't just about what version we should use in our images, it was how we should deal with the bad interactions with older versions of nftables. It's one thing to say "nftables kube-proxy will not work when using nft < 1.0.1", but it's another thing to say "you are not allowed to run nftables kube-proxy on hosts where anybody is using nft < 1.0.1 (because otherwise kube-proxy will break other users of nft)". There is no reliable way for us to figure out the version of nft on the host from within the kube-proxy container. The best we could do would be to assume that the nft and kernel versions are roughly contemporary, and use the kernel version as a proxy. nft 1.0.1 was contemporary with kernel 5.16, and nft 1.0.0 was contemporary with 5.13. If we require kernel 5.13 / nft 1.0.1, we can support distros newer than mid 2021: RHEL 9 (kernel 5.14 + backports / nft 1.0.4) Debian 12 / bookworm / newstable (kernel 6.1 / nft 1.0.6) Ubuntu 22.04LTS (kernel 5.15 or newer / nft 1.0.2) (Not sure about SLES?)

Recent Fedora and other non-LTS distros Whereas our current plan of requiring kernel 5.4 / nft 0.9.2 means "newer than late 2019" and adds: RHEL 8 (kernel 4.18 + many backports / originally nft 0.9.3, later 1.0.4) Debian 11 / bullseye / oldstable (kernel 5.10 / nft 0.9.8) Ubuntu 20.04LTS (kernel 5.4 or newer / nft 0.9.3) (Not sure about SLES?) OK, so first decision: do we feel like we need to support running nftables kube-proxy on RHEL 8 / Debian 11 / Ubuntu 20.04LTS, or are we OK with only supporting the newer distros? For comparison, seems to be leaning toward 4.18/4.19 (from late 2018) as the minimum supported kernel for kubernetes as a whole, so requiring kernel 5.13 for nftables would add several more years before we could really even consider making it the default backend for kube-proxy. (We could also have kube-proxy require the newer versions by default, but have a config option to allow the older versions, and document exactly what can go wrong with that.) If we do want to still support the older distros (either unconditionally or via config option), there are two more questions: we decouple our kernel and nft requirements, and say that kube-proxy requires nft = 1.0.1 regardless of what kernel you're using? (I think "yes"?) we want to do some sort of workaround for the segfault bug (either changing the endpoint rules, or shipping the kube-proxy image with nft == 1.0.2 rather than whatever-version-is-currently-in-debian)? (If we had a config option to allow older kernels, we could make that option also control whether we used vmap-style or iptables-style endpoint rules?)

OK, leaning toward: Docs: We document that nftables kube-proxy requires nftables = 1.0.1, and recommends a kernel = 5.13 We also document that, currently, it can be used on systems with kernels as old as 5.4 if you set the option in the kube-proxy config, but (a) this will likely cause problems with other nftables users on the system (with details), and (b) this option is intended to go away at some point as the LTS distros with older kernels become unsupported. Code: We add to We check the nft client version (via a new knftables API), and if it's < 1.0.1, we error out. We check the kernel version (via ) and if it's < 5.13: if is (the default) we error out if is : if an with a expression passes, we continue, after warning that we might break other users on the system if the fails, we error out because the kernel is too oldWe don't change our rules or change the version of nft in our images, because the problems will only affect users who have explicitly opted in.
oh, and CI apparently has a 5.15 kernel, so we won't need to update the kind job config
I envision we'll hit more of these things in the future, so it will be nice you implement it, in a modular way, kind of the iptables runner that has all these different behaviors hidden from the users
The idea was that for things that affect knftables's own functioning, or its public API, it would check those things itself and hide the details from the caller. But the caller would be expected to use to figure out things like "can I use in a rule", since knftables can't just abstract around that (in general). (Right now the only thing knftables abstracts around is whether table/chain/set/map comments are supported; you're always allowed to specify them in the API, but .) I was originally thinking that the "require nft 1.0.1" restriction would be at the kube-proxy level rather than the knftables level, but I'm thinking now it would make more sense to require that for all knftables users.


Question: Via a _label selector_, the client/user can identify a set of objects. The label selector is the core grouping primitive in Kubernetes. The API currently supports two types of selectors: _equality-based_ and _set-based_. <del> A label selector can be made of multiple _requirements_ which are comma-separated. In the case of multiple requirements, all must be satisfied so comma separator acts as an AND logical operator. </del> <ins> A label selector can be made of multiple _requirements_ which are comma-separated. In the case of multiple requirements, all must be satisfied so the comma separator acts as an _AND_ logical operator. </ins> An empty label selector (that is, one with zero requirements) selects every object in the collection.

The suggests we support both "equality-based" and "set-based" label selectors, but actually we only support "==" selector right now. This is confusing. People are asking questions on IRC. We should update the doc in both github and
cc
We support the set-based selectors for filtering GET/watch requests. We support equality-based selectors, represented as maps, for API resources, such as ReplicationController, Service, and Pod nodeSelector.
Even if it's technically correct people are getting confused, so we need to fix the documentation.
Ref
sorry I see this only now. I'm ok to better explain the difference between set-based and equality based but I would need of some example of questions on IRC. May you have some of them? Otherwise we may clearly write that kubectl <command-l <selectorsupports set-based and equality based selector while API resources supports only equality-based.
I was thinking of making the difference between selectors in kubectl and in API clear.
cool, I can work on this if you've not already started.
By all means - please do:)
:)


Question: - name: kube-dns-config mountPath: /kube-dns-config - name: dnsmasq <del> image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2 </del> <ins> image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.3 </ins> livenessProbe: httpGet: path: /healthcheck/dnsmasq

A number of addon images have CVEs that have been fixed in the upstream base images. We should update these images before the final 1.7 release. cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/cc
cc
The new image appears to have removed . I'm not sure if this was expected, but it caused at least fluentd to stop working on COS (). This might also affect kubeproxy, which uses (a derivative of ).
also the metadata-proxy, prometheus-to-sd, the fluentd-event-exporter, and ip-masq-agent.
anything more to do for this issue?
yes, I need to get merged and update those images again. I also need to bump a few other images (dns and ip-masq-agent).
should be the last update needed.


Question: cmdutil.CheckErr(fmt.Errorf("output must be one of '' or 'name': %v", outputFormat)) } if !supportedOutputTypes.Has(outputFormat) { <del> fmt.Fprintf(out, "--output %v is not available in kubectl config get-contexts; reseting to default output format", outputFormat) </del> <ins> fmt.Fprintf(out, "--output %v is not available in kubectl config get-contexts; resetting to default output formatn", outputFormat) </ins> cmd.Flags().Set("output", "") } cmdutil.CheckErr(options.Complete(cmd, args, out))

When specifying an invalid output format, the error message and the table headers are printed on the same line, producing hard-to-read output:


Question: print('File %s is missing the year' % filename, file=verbose_out) return False <del> # Replace all occurrences of the regex "2017|2016|2015|2014" with "YEAR" </del> <ins> # Replace all occurrences of the regex "2014|2015|2016|2017|2018" with "YEAR" </ins> p = regexs["date"] for i, d in enumerate(data): (data[i], found) = p.subn('YEAR', d)

Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.