Question: proxier = proxierIPVS serviceEventHandler = proxierIPVS endpointsEventHandler = proxierIPVS <del> klog.V(0).Info("Tearing down inactive rules.") // TODO this has side effects that should only happen when Run() is invoked. userspace.CleanupLeftovers(iptInterface) iptables.CleanupLeftovers(iptInterface) </del> } else { klog.V(0).Info("Using userspace Proxier.") // This is a proxy.LoadBalancer which NewProxier needs but has methods we don't need for

What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

What happened: When restarting kube-proxy in iptables mode, there will be several seconds of timeouts because it flushes the KUBE-SERVICES nat chain and several others. What you expected to happen: Restarting kube-proxy should not impact traffic in any way. How to reproduce it (as minimally and precisely as possible): an HTTP server behind a ClusterIP service an http client in a loop making requests to the HTTP server -15 the kube-proxy process on the client node timeout for a few seconds until kube-proxy re-syncs the iptables rules Anything else we need to know?: Running with makes it pretty clear what's happening: it says it's cleaning up inactive rules, but these are crucial to the iptables proxier. You can trivially reproduce by just running: Environment: Kubernetes version (use ): 1.10.11 Cloud provider or hardware configuration: amazon OS (e.g: ): custom Kernel (e.g. ): 4.14.77-hs623.el6.x86_64 Install tools: custom Others: iptables 1.6.2 We're running kube-proxy 1.10.11, but I've confirmed this issue with kube-proxy 1.13.4 too
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Possibly a bug in the proxy. It flushes iptabes if the proxier is able to use IPVS which seems like the wrong behavior to me if there are shared tables/chains between the IPVS and iptables proxier. One workaround I can think of right now is to unload the ipvs kernel module to skip this check.
/assign
oh, that's an interesting workaround, I'll give that a shot, thanks!
np, I'll try to work on the bug fix in the meantime!
/triage unresolved
note that unloading the ipvs module is not actually enough of a workaround because kube-proxy will load the kernel module if it sees that it is available. If I unload the ipvs module and then rename to something else in then it does the right thing. Dug slightly further: the way it probes for modules is by actually running modprobe.

So the simplest hack that allows us to keep ipvs loaded for other things on the system is to put a modprobe script on its PATH that just always exits 1 for the ip_vs modules. (We can stop mounting the modules dir into the kube-proxy container in our kubernetes clusters, but we also run kube-proxy on non-kubernetes nodes via the init system, so the PATH hack works well enough there). This should hold us over fine until we get to a version with your fix in it.
Good to know, thanks for sharing!
/assign
Quick update on this issue from today's SIG Network call: we're going to try to get rid of the automatic proxy clean up altogether for v1.14.1 since this is considered a bug. is working on the KEP & implementation.


Question: RecordFlags *genericclioptions.RecordFlags PrintFlags  *genericclioptions.PrintFlags <del> DeleteFlags *delete.DeleteFlags </del> <ins> DeleteFlags *cmddelete.DeleteFlags </ins> FieldManager   string Selector       string

As per , I'm trying to remove CPU limits. But the behaviour is different depending whether the ResourceQuota exists or not: I would expect to be idempotent, and for its behaviour to result in the same end configuration regardless of the initial configuration. a new namespace named this command: that the ResourceQuota is created and is set to it again that is removed each subsequent time it's run, as long as resourcequota/compute-resource exists resourcequota/compute-resource again that the ResourceQuota is created and is set to again, and so forth the exact same behaviour if is changed to No response <details</details<detailsOpenShift 4.13.12 </details<details</details<details</details<details</details<details</details
/sig node
Tried to replicate the issue and observed the same behaviour.
/remove-sig node /sig api-machinery /cc
What's the goal of applying here? What are you trying to achieve? I agree that this is not the right behavior though.
I'm trying to remove a previously set property. I cannot remove the ResourceQuota altogether because it contains other properties (which I left out of my example) that I do not wish to remove.
But 0 is the same as unset, isn't it?
No, if I set it to 0 then any deployment with set to higher than 0 will fail to create any pods because it will exceed the quota.
/triage accepted /cc
/assign
Hello, I'm having a similar issue. Here's my use case: We use Flux to deploy things on our cluster - in particular, we use the component and set chart values in . Some of the charts we deploy have default values we want to remove, so we want to set a value to Editing an HelmRelease with this kind of ajustments drops the null keys, so default values end up being setup again and generated manifests are invalid :( Let me know if you need anything else to keep this issue moving!
I think the correct behavior is to set the struct to the default value when or , because in the standard library does this as well, and I think we expect the result to be as similar as possible to . So our real issue here is that we didn't set the to its default value when patch.
Based on the above, I think this question is not related to the current issue.

Tracing it back to , maybe the current result is what we want. The explicit null values means that we want to remove it in patch. As mentioned , our behavior in create is also as expected. I'm not quite sure if we actually want to fix it, and how /cc I hope to get your advice as you are the implementer of
I think client-side on create should do what on update will accomplish, namely, explicit nulls in the manifest are treated as the removal of the attributes they are values of. Proof of concept opened at , I'd be happy for someone to pick that up, add test coverage, and work through any failing tests that change uncovers
Thank you, but I still have a question. If we solve the difference between create and patch behaviors in ApiServer on the client side, will third-party callers still have the same problem? Or should we not cover this scenario?
I would consider changing behavior of null in a create request server-side to be a compatibility break. values are defined for client-side apply to mean "remove the key associated with this value". I think fixing the client-side apply creation case to have the same semantic is the most narrowly scoped fix we can make while preserving compatible server-side behavior generally.
Okay, I got it, I'll follow up with


Question: visibility = ["//visibility:public"], deps = [ "//pkg/scheduler/algorithm/predicates:go_default_library", <del> "//pkg/scheduler/framework/plugins/migration:go_default_library", </del> "//pkg/scheduler/framework/v1alpha1:go_default_library", "//pkg/scheduler/nodeinfo:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

/sig scheduling /priority important-soon See as an example. Part of /assign
GitHub didn't allow me to assign the following users: angao. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig scheduling /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thx, I will do it soon.


Question: }, []string{NodeLabelKey}, ) <del> // ContainersPerPodCount is a Counter that tracks the number of containers per pod. </del> <ins> // ContainersPerPodCount is a Histogram that tracks the number of containers per pod. </ins> ContainersPerPodCount = metrics.NewHistogram( &metrics.HistogramOpts{ Subsystem:      KubeletSubsystem, Name:           "containers_per_pod_count", Help:           "The number of containers per pod.", <del> Buckets:        metrics.DefBuckets, </del> <ins> Buckets:        metrics.ExponentialBuckets(1, 2, 5), </ins> StabilityLevel: metrics.ALPHA, }, )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There is an metric in kubelet, which is used to track the number of containers per pod. The current metric data format is The number of containers in the pod will definitely not be less than 1, so the bucket is set to have optimized space What you expected to happen: Would it be better if optimized to the following bucket How to reproduce it (as minimally and precisely as possible): curl kubelet metrics endpoint Anything else we need to know?: Environment: Kubernetes version (use ): 1.19,1.20 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /area kubelet
/assign
/triage accepted


Question: if !isHardEvictionThreshold(thresholdToReclaim) { gracePeriodOverride = m.config.MaxPodGracePeriodSeconds } <del> message, annotations := evictionMessage(resourceToReclaim, pod, statsFunc) </del> <ins> message, annotations := evictionMessage(resourceToReclaim, pod, statsFunc, thresholds, observations) </ins> var condition *v1.PodCondition if utilfeature.DefaultFeatureGate.Enabled(features.PodDisruptionConditions) { condition = &v1.PodCondition{

when the pod is evicted due to low imagefs.available, the prompt is difficult to understand. it showed: But the container was evicted not by exceeding its request of 0 ephemeral-storage, but due to reaching the evictionHard of the node. The correct prompt message points out the reason for the pod eviction. don't print "which exceeds its request of 0.", because the container was evicted not by exceeding its request of 0. % oc label mcp worker custom-kubelet-eviction=small-pods evictionHard by creating a kubeletconfig % oc create -f custom-kubelet- custom-kubelet- several minutes, the mcp worker finish rolling out to the latest mc a pod % oc create -f hello-pod- hello-pod- a large file in container, make imagefs.available < 90Gi (the current available disk size is 111G ) command terminated with exit code 137 the node status the pod status % oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-pod-2 0/1 ContainerStatusUnknown 1 7m58s 10.131.0.7 ip-10-0-213- <none<none% oc get pod hello-pod-2 -o yaml No response <details</details<detailsaws </details<details</details<details</details<details</details<details</details
/sig node
/assign
/assign
/triage accepted


Question: if err != nil { return nil, err } <del> domainNameLabel := getPublicIPDomainNameLabel(service) pip, err := az.ensurePublicIPExists(service, pipName, domainNameLabel, clusterName, shouldPIPExisted) </del> <ins> domainNameLabel, found := getPublicIPDomainNameLabel(service) pip, err := az.ensurePublicIPExists(service, pipName, domainNameLabel, clusterName, shouldPIPExisted, found) </ins> if err != nil { return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: sourceRangeCidrs = append(sourceRangeCidrs, "0.0.0.0/0") } <del> err = c.updateInstanceSecurityGroupsForNLB(loadBalancerName, instances, sourceRangeCidrs, v2Mappings) </del> <ins> err = c.updateInstanceSecurityGroupsForNLB(loadBalancerName, instances, subnetCidrs, sourceRangeCidrs, v2Mappings) </ins> if err != nil { klog.Warningf("Error opening ingress rules for the load balancer to the instances: %q", err) return nil, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws


Question: Ports: []v1.ServicePort{ { Name:       "http", <ins> Protocol:   v1.ProtocolTCP, </ins> Port:       80, TargetPort: intstr.FromInt(8080), }, }, <ins> SessionAffinity: v1.ServiceAffinityNone, </ins> }, } By(fmt.Sprintf("Trying to create service %q in namespace %q", service.Name, namespace))

FYI
this should be a pretty simple thing to fix, if you have the time?
I am able to send a quick fix by EOD.


Question: controllers[e.Regarding.Name] = e.ReportingController wg.Done() }) <ins> if err != nil { t.Fatal(err) } </ins> defer stopFn() // Run scheduler.

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // Set this weight large enough to override all other priority functions. // TODO: Figure out a better way to do this, maybe at same time as fixing #24720. <del> scheduler.RegisterPriorityMapReduceFunction(priorities.NodePreferAvoidPodsPriority, priorities.CalculateNodePreferAvoidPodsPriorityMap, nil, 10000) </del> <ins> scheduler.RegisterPriorityMapReduceFunction(priorities.NodePreferAvoidPodsPriority, nil, nil, 10000) </ins> // Prioritizes nodes that have labels matching NodeAffinity scheduler.RegisterPriorityMapReduceFunction(priorities.NodeAffinityPriority, nil, nil, 1)

/sig scheduling /priority important-soon See as an example. Part of This shouldn't need a PostFilter Plugin, it is not worth caching the result of metav1.GetControllerOf. /assign


Question: ``` cd kubernetes/.git/hooks/ <del> ln -s ../../hooks/prepare-commit-msg . ln -s ../../hooks/commit-msg . </del> <ins> ln -s ../../hooks/pre-commit . </ins> ``` ## Unit tests

Multiple, for example: Since See above The test tolerations are now to strict: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think it's from if you check the cri-o versions the succeessful one was and the failing was
Working on a fix in
cc
/reopen need to bump in kube as well to fully get the fix
Reopened this issue. <detailsIn response to : /reopen </details


Question: MASTER_SIZE=g1-small MINION_SIZE=g1-small NUM_MINIONS=${NUM_MINIONS:-2} <ins> MINION_DISK_TYPE=pd-standard MINION_DISK_SIZE=10GB </ins> # TODO(dchen1107): Filed an internal issue to create an alias # for containervm image, so that gcloud will expand this # to the latest supported image.

The node disk type should be an option in the cluster/gce/config-*.sh files. The nodes get created here:
And also the disk size.
How to do this for aws ? I am stuck at 10GB minion
It . A pull request adding it would be very welcome!
please check this thread
Hello there, can I do something to make this available in the command ? I'd like to us an SSD drive :)


Question: svc.Spec.ExternalName = "google.com" return svc } <ins> func TestEndpointTruncate(t *testing.T) { // Disable ServiceAccount admission plugin as we don't have serviceaccount controller running. server := kubeapiservertesting.StartTestServerOrDie(t, nil, framework.DefaultTestServerFlags(), framework.SharedEtcd()) defer server.TearDownFn() client, err := clientset.NewForConfig(server.ClientConfig) if err != nil { t.Fatalf("Error creating clientset: %v", err) } informers := informers.NewSharedInformerFactory(client, 0) tCtx := ktesting.Init(t) epController := endpoint.NewEndpointController( tCtx, informers.Core().V1().Pods(), informers.Core().V1().Services(), informers.Core().V1().Endpoints(), client, 0) // Start informer and controllers informers.Start(tCtx.Done()) go epController.Run(tCtx, 1) // Create namespace ns := framework.CreateNamespaceOrDie(client, "test-endpoints-truncate", t) defer framework.DeleteNamespaceOrDie(client, ns, t) // Create a pod with labels basePod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "test-pod", Labels: labelMap(), }, Spec: v1.PodSpec{ NodeName: "fake-node", Containers: []v1.Container{ { Name:  "fakename", Image: "fakeimage", Ports: []v1.ContainerPort{ { Name:          "port-443", ContainerPort: 443, }, }, }, }, }, Status: v1.PodStatus{ Phase: v1.PodRunning, Conditions: []v1.PodCondition{ { Type:   v1.PodReady, Status: v1.ConditionTrue, }, }, PodIP: "10.0.0.1", PodIPs: []v1.PodIP{ { IP: "10.0.0.1", }, }, }, } // create 1001 Pods to reach endpoint max capacity that is set to 1000 allPodNames := sets.New[string]() baseIP := netutils.BigForIP(netutils.ParseIPSloppy("10.0.0.1")) for i := 0; i < 1001; i++ { pod := basePod.DeepCopy() pod.Name = fmt.Sprintf("%s-%d", basePod.Name, i) allPodNames.Insert(pod.Name) podIP := netutils.AddIPOffset(baseIP, i).String() pod.Status.PodIP = podIP pod.Status.PodIPs[0] = v1.PodIP{IP: podIP} createdPod, err := client.CoreV1().Pods(ns.Name).Create(tCtx, pod, metav1.CreateOptions{}) if err != nil { t.Fatalf("Failed to create pod %s: %v", pod.Name, err) } createdPod.Status = pod.Status _, err = client.CoreV1().Pods(ns.Name).UpdateStatus(tCtx, createdPod, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Failed to update status of pod %s: %v", pod.Name, err) } } // Create a service associated to the pod svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-service", Namespace: ns.Name, Labels: map[string]string{ "foo": "bar", }, }, Spec: v1.ServiceSpec{ Selector: map[string]string{ "foo": "bar", }, Ports: []v1.ServicePort{ {Name: "port-443", Port: 443, Protocol: "TCP", TargetPort: intstr.FromInt32(443)}, }, }, } _, err = client.CoreV1().Services(ns.Name).Create(tCtx, svc, metav1.CreateOptions{}) if err != nil { t.Fatalf("Failed to create service %s: %v", svc.Name, err) } var truncatedPodName string // poll until associated Endpoints to the previously created Service exists if err := wait.PollUntilContextTimeout(tCtx, 1*time.Second, 10*time.Second, true, func(context.Context) (bool, error) { podNames := sets.New[string]() endpoints, err := client.CoreV1().Endpoints(ns.Name).Get(tCtx, svc.Name, metav1.GetOptions{}) if err != nil { return false, nil } for _, subset := range endpoints.Subsets { for _, address := range subset.Addresses { podNames.Insert(address.TargetRef.Name) } } if podNames.Len() != 1000 { return false, nil } truncated, ok := endpoints.Annotations[v1.EndpointsOverCapacity] if !ok || truncated != "truncated" { return false, nil } // There is only 1 truncated Pod. truncatedPodName, _ = allPodNames.Difference(podNames).PopAny() return true, nil }); err != nil { t.Fatalf("endpoints not found: %v", err) } // Update the truncated Pod several times to make endpoints controller resync the service. truncatedPod, err := client.CoreV1().Pods(ns.Name).Get(tCtx, truncatedPodName, metav1.GetOptions{}) if err != nil { t.Fatalf("Failed to get pod %s: %v", truncatedPodName, err) } for i := 0; i < 10; i++ { truncatedPod.Status.Conditions[0].Status = v1.ConditionFalse truncatedPod, err = client.CoreV1().Pods(ns.Name).UpdateStatus(tCtx, truncatedPod, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Failed to update status of pod %s: %v", truncatedPod.Name, err) } truncatedPod.Status.Conditions[0].Status = v1.ConditionTrue truncatedPod, err = client.CoreV1().Pods(ns.Name).UpdateStatus(tCtx, truncatedPod, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Failed to update status of pod %s: %v", truncatedPod.Name, err) } } // delete 501 Pods for i := 500; i < 1001; i++ { podName := fmt.Sprintf("%s-%d", basePod.Name, i) err = client.CoreV1().Pods(ns.Name).Delete(tCtx, podName, metav1.DeleteOptions{}) if err != nil { t.Fatalf("error deleting test pod: %v", err) } } // poll until endpoints for deleted Pod are no longer in Endpoints. if err := wait.PollUntilContextTimeout(tCtx, 1*time.Second, 10*time.Second, true, func(context.Context) (bool, error) { endpoints, err := client.CoreV1().Endpoints(ns.Name).Get(tCtx, svc.Name, metav1.GetOptions{}) if err != nil { return false, nil } numEndpoints := 0 for _, subset := range endpoints.Subsets { numEndpoints += len(subset.Addresses) } if numEndpoints != 500 { return false, nil } truncated, ok := endpoints.Annotations[v1.EndpointsOverCapacity] if ok || truncated == "truncated" { return false, nil } return true, nil }); err != nil { t.Fatalf("error checking for no endpoints with terminating pods: %v", err) } } </ins>

This issue was supposed to have fixed the issue where endpoint stay out of sync I also wrote a small script which would get me the out of sync endpoints compared to the endpointslices I expect the endpoints to eventually sync and reflect the most upto date information. I have just deployed the newer patch to our cluster and that has resulted in endpoints never ending up being updated if the status goes out of sync. No response Client Version: v1.29.7 Kustomize Version: v5.0.4-0.- Server Version: v1.29.7 <details</detailsalmalinux-9 <details</detailscri-o No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
from the logs, endpoint controller was retrying test1/test-qa in the expected way. It has retried 5 times and the next retry should be after 320ms, which should long enough for the informer cache to get the updated endpoints. If there is no more logs related to the failure, I think it may have synced it successfully. The python script used to detect the failure seems capturing diff between endpoints and endpointslice at a moment, but that could happen if there is any Pod change in the cluster, and technically the script is comparing endpoints of one moment with endpointslice of another moment. So did you see the endpoints not being synced after endpoint controller stopped retrying it?
Yeah i still see it and see similar messages continuously. Let me post some additional log messages after the one i posted above.
Observed the same issue in 1.29 and 1.30. I keep seeing the same error then new pods were behind a service. And the endpoint never catch up after all pods are settled.
could you provide complete log of kube-controller-manager and the yamls of the related endpoints and endpointslice?
/assign Are you able to triage this since you are already looking at this?

Yes i can get that over by EOD today
Sure, I'm waiting for more logs and the content of the endpoints to understand how it happened. Currently there are only retry logs which is expected in some cases.
We found a customer that triggered an edge case in your change of , which will cause Endpoints never updated. They have a mutatingWebhook on Endpoints object that automatically injects a label, let's say . syncServices, the endpoints controller , and they won't match since Endpoints have the extra "dummy-k: value". endpoints controller will by removing the label. mutatingWebhook will automatically adds back the label as part of the update request. From the APIServer's POV, the Endpoints object don't have any changes, thus the update request will succeed but returns the same resourceVersion as before. change in will mark that resourceVersion as stale. future pods events won't be handled correctly due to the step didn't result in a new resourceVersion. I think we should fix this by check the resulting resourceVersion from the update request, and only mark the old resourceVersion as stale when a new resourceVersion is generated. Not sure whether this is the same case others in this thread encountered :D
thanks for explaining the use case and the analysis. I did analyze the code after the issue was opened and suspected the only possibility is the returned resource version somehow equals to the one in the request but didn't know how it could happen after it has checked the desired endpoints and the current endpoints are different: Now I understand this could happen when the endpoints appear different on client side but are mutated to be same on server side. Do you plan to create a PR since you have proposed a fix? could you confirm if this is also your case, i.e. there is a mutatingWebhook for endpoints objects that could make the endpoints different from what endpoints controller expects?
Yeah, i can do a fix for this
- can we assign to you?
Sure, I'm already assigned. Given that we've had three user reports on this issue, I think it's important to address it as soon as possible. Could you please let me know when you plan to create the PR? If you're currently occupied, I'm happy to handle it and submit the PR.
Sorry, were occupied by some internal incidents last week. Just created a WIP PR for it.

Will test it in kops for this specific webhook scenario in the following two days and and remove the WIP flag
/assign


Question: monitoring-heapster-controller-fplln         10.244.0.4          heapster                kubernetes/heapster:v0.8            kubernetes-minion-2il2.c.kubernetes-user2.internal/130.211.155.16   kubernetes.io/cluster-service=true,name=heapster,uses=monitoring-influxdb   Running             5 hours monitoring-influx-grafana-controller-0133o   10.244.3.4          influxdb                kubernetes/heapster_influxdb:v0.3   kubernetes-minion-kmin.c.kubernetes-user2.internal/130.211.173.22   kubernetes.io/cluster-service=true,name=influxGrafana                       Running             5 hours grafana                 kubernetes/heapster_grafana:v0.4 <del> ``` </del> <ins> ``` </ins>  No newline at end of file

is out of date. Please update for the current location/contents of the file and v1beta3. Once is submitted we should also update the examples to show to to curl using bearer tokens instead of basic auth credentials.
We're working on scrubbing the docs and examples of references to v1beta1 () prior to deleting that API, so it would be good to have this in soon.
: let me know if you want to punt this back to me -- now that I have some clue again about how to access an cluster (assuming I can create a cluster!).
Punting back to you. Getting the auth credentials is accomplished via But I think you have a better idea how to curl with token auth than I do.


Question: cmd.AddCommand(NewCmdResources(f, out, err)) cmd.AddCommand(NewCmdSelector(f, out)) cmd.AddCommand(NewCmdSubject(f, out, err)) <ins> cmd.AddCommand(NewCmdServiceAccount(f, out, err)) </ins> return cmd }

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): HEAD Now that RBAC is going to GA, using ServiceAccounts is getting more and more important. Still a lot of users run their apps/addons/whatever in the SA. In order for the admin to be able to change ServiceAccount easily after an application has been installed (which supposedly is crashlooping due to insufficient privs), a command could exist. I know I can do this: but for new users to Kubernetes, it's non-obvious. Proposed command: Tracking issue: WDYT? This would be a good way for a new contributor to get aquainted with the code by contributing this feature.
Is anyone working on this? If not I would like to take this up.
Nobody is working on this. I would be super happy if you could contribute this feature. Be aware of that there isn't consensus around this proposal yet (I proposed it yesterday, saturday, so people haven't had time to catch up on it) However, I do think that this is in line with , so feel free to start working on this if you want to ;) Also, don't hesitate to ask for help if you don't know how to get started and we'll help out! Thanks!
I'm in favor of this command. ?
Thanks, I will start working on this.
Is the SA on a Pod mutable or immutable? What about DaemonSet and StatefulSet? Do those have SA as well? Is there a way to do this so that it dynamically supports anything with a PodTemplate? Maybe this should be done as a subresource that can be dynamically discovered by kubectl using the discovery service.
Maybe helpful:
Pod is immutable. All the other objects (with the exception of stateful set) are mutable.
That pattern is the same for all the things that set aspects of a pod spec (except image, which is also mutable in pods)
Statefulsets are going to loosen their validation soon I expect.
This sounds good to me. just go ahead, I will help review this. Thanks ~
The outout of this command kubectl get rc/nginx-controller -o=yaml , has service acccount at the following path '' , and the corresponding pod has service account at 'spec.serviceAccountName'.

Both refer to the 'serviceAccountName' field of PodSpec struct but can be made to be different by patching the ReplicationController resource. How does ReplicationController and Pod use serviceAccountName differently? Shouldn't the ReplicationControllerSpec have a serviceAccountName field?
ptal, I think you are more familiar with ReplicationController implementation than me.
ReplicationController contains a pod spec object, which contains a serviceAccountName.
Hi , how to go about writing e2e test cases for this. My development is mostly complete.
I wouldn't mind if you posted the PR now and then we'd see what e2e test makes sense to have. In order to move forward...
what are the next steps?


Question: listener    Listener httpFactory HTTPServerFactory <del> lock     sync.Mutex </del> <ins> lock     sync.RWMutex </ins> services map[types.NamespacedName]*hcInstance }

What would you like to be : Use read lock in ServeHTTP Why is this needed: Currently ServeHTTP (in ) takes the write lock: Since it only reads from h.hcs.services, holding read lock should suffice.
cc
/sig network
ping
looks good to me. Thanks!
Can you give /lgtm ?
ping


Question: // the list automatically by PodDisruptionBudget controller after some time. // If everything goes smooth this map should be empty for the most of the time. // Large number of entries in the map may indicate problems with pod deletions. <ins> // +optional </ins> DisruptedPods map[string]metav1.Time // Number of pod disruptions that are currently allowed.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PodDisruptionBudget provides inappropriate message and cannot be updated as is required filed. What you expected to happen: works fine for editing labels under . Also, produce correct error message when edit field. How to reproduce it (as minimally and precisely as possible): Then, we will get following error after step-2. When edit , it should provide correct message as .When edit especially labels under metadata, they should be updated without error. Anything else we need to know?: According to the code comment, is valid. Environment: Kubernetes version (use ): build on
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale there is a PR open


Question: [parameter(Mandatory = $false)] [System.Collections.IDictionary]$Headers = @{} ) <ins> # Load the System.Net.Http assembly if it's not loaded yet. if ("System.Net.Http.HttpClient" -as [type]) {} else { Add-Type -AssemblyName System.Net.Http } </ins> $timeout = New-TimeSpan -Minutes 5 try {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: GCE Windows cluster bring up fails on some versions of Windows. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This is a regression caused by Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign jeremyje /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign pjh
/unassign pjh


Question: Release artifacts are [signed](https://github.com/kubernetes/enhancements/issues/3031) using [cosign](https://github.com/sigstore/cosign) signatures <del> and there is experimental support for [verifying image signatures](/docs/tasks/administer-cluster/verify-signed-images/). </del> <ins> and there is experimental support for [verifying image signatures](https://kubernetes.io/docs/tasks/administer-cluster/verify-signed-images/). </ins> Signing and verification of release artifacts is part of [increasing software supply chain security for the Kubernetes release process](https://github.com/kubernetes/enhancements/issues/3027). ### OpenAPI v3

has some broken links. These are relative links that work OK in the release blog article but need to be absolute hyperlinks in the release notes. Links in release notes work. Visit Click on all the links. Specifically, “this guide” in (there are some more, this is one example) No response v1.24.0 n/a n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release and IMO /priority important-soon
Hi Tim, I can help fix this /assign


Question: return buffer.String() } <ins> // safeSort sort input strings without modification func safeSort(in []string) []string { if sort.StringsAreSorted(in) { return in } out := make([]string, len(in)) copy(out, in) sort.Strings(out) return out } </ins> // Add adds requirements to the selector. It copies the current selector returning a new one func (lsel internalSelector) Add(reqs ...Requirement) Selector { var sel internalSelector

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After creating a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values, pods get created on the appropriate nodes—and, fortunately, only on the appropriate nodes—but the pods get terminated and deleted quickly before they can start running. The cycle takes about ten seconds from pod creation through completion of deletion; the pod appears to be deleted within one second after its creation, but it takes about ten seconds for the deletion to complete. If I vary the DaemonSet's scheduling predicates, I find the following work to get pods running successfully: Use to choose a specific node. Use with an operator of "Exists" or "NotIn" to choose some nodes. Use with an operator of "In," but with only one value to choose some nodes. Use with a label that is not applied by the cloud provider. If I come up with my own label such as "special-hardware" and use kubectl label node to apply it, and use that label as a match expression key, it seems to work fine. It's the dynamically applied labels like "" and "" that trigger this problem. Without the above concessions, if I use with an operator of "In," but with more than one value, the pods don't start correctly. Note that this behavior is a regression from Kubernetes version 1.10.4. The same configuration works as intended in a cluster running that earlier version. What you expected to happen: After creating the DaemonSet, pods would start successfully on the nodes with label values matching one of the DaemonSet's candidates values. How to reproduce it (as minimally and precisely as possible): a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values. that at least one node has a label with a value that matches a member of that candidate set. kubectl get pods -o wide to observe that there are pods created on behalf of the DaemonSet on nodes that match the predicate, but they have status "Terminating." By watching the pods, you can see new pods arrive with status "Pending," then "ContainerCreating," then "Terminating," which they'll retain until deletion completes and a replacement arrives. a similar pod directly, without a supervising DaemonSet.

If the "" label is inconvenient within your cluster, instead consider using "" and a few hostnames as the candidate values. that one pod starts running successfully on one of the nodes that match that the predicate. that lone pod (for clarity), and vary the predicate on the DaemonSet to see which changes allow pods to start running successfully. Try each of these individually: Remove one of the values in the sequence. That is, leave only one candidate value. Change the operator to "NotIn," and adjust the values to select some subset of the nodes. Alternately, try "Exists" with no values. Remove the node affinity stanza's and add a node selector in its place: Anything else we need to know?: Here is the DaemonSet object as captured via kubectl get daemonset -o yaml: Here is one of the pods created on behalf of the DaemonSet: Here is one of the nodes on which pods like these should run: There some preceding discussion in the "sig-node" channel of the "Kubernetes" Slack team . Possibly related issues: , Possibly related PRs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 (g3.8xlarge instance, but other instance types exhibit the same behavior) OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm init, kubeadm join
/sig node /sig apps
To work around this problem, I found that I can use multiple match expressions, with one candidate value in each. This works because sibling match expressions are a disjunction.
if the kubelet is rejecting the pod, then this belongs to sig-node it might be reusing logic from the scheduler library, so looping in sig-scheduling for reference: /sig scheduling
I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. /assign
GitHub didn't allow me to assign the following users: Huang-Wei. Note that only and repo collaborators can be assigned. <detailsIn response to : I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. </details
I've found the root cause, and should be able to send out a PR next week.


Question: if kubeClient == nil { return nil, fmt.Errorf("glusterfs: failed to get kube client to initialize mounter") } ep, err := kubeClient.Core().Endpoints(ns).Get(epName, metav1.GetOptions{}) <del> if err != nil && errors.IsNotFound(err) { claim := spec.PersistentVolume.Spec.ClaimRef.Name checkEpName := dynamicEpSvcPrefix + claim if epName != checkEpName { return nil, fmt.Errorf("failed to get endpoint %s, error %v", epName, err) } glog.Errorf("glusterfs: failed to get endpoint %s[%v]", epName, err) if spec != nil && spec.PersistentVolume.Annotations[volumehelper.VolumeDynamicallyCreatedByKey] == heketiAnn { class, err := volutil.GetClassForVolume(plugin.host.GetKubeClient(), spec.PersistentVolume) if err != nil { return nil, fmt.Errorf("glusterfs: failed to get storageclass, error: %v", err) } cfg, err := parseClassParameters(class.Parameters, plugin.host.GetKubeClient()) if err != nil { return nil, fmt.Errorf("glusterfs: failed to parse parameters, error: %v", err) } scConfig := *cfg cli := gcli.NewClient(scConfig.url, scConfig.user, scConfig.secretValue) if cli == nil { return nil, fmt.Errorf("glusterfs: failed to create heketi client, error: %v", err) } volumeID := dstrings.TrimPrefix(source.Path, volPrefix) volInfo, err := cli.VolumeInfo(volumeID) if err != nil { return nil, fmt.Errorf("glusterfs: failed to get volume info, error: %v", err) } endpointIPs, err := getClusterNodes(cli, volInfo.Cluster) if err != nil { return nil, fmt.Errorf("glusterfs: failed to get cluster nodes, error: %v", err) } // Give an attempt to recreate endpoint/service. _, _, err = plugin.createEndpointService(ns, epName, endpointIPs, claim) if err != nil && !errors.IsAlreadyExists(err) { glog.Errorf("glusterfs: failed to recreate endpoint/service, error: %v", err) return nil, fmt.Errorf("failed to recreate endpoint/service, error: %v", err) } glog.V(3).Infof("glusterfs: endpoint/service [%v] successfully recreated ", epName) } else { return nil, err } </del> <ins> if err != nil { glog.Errorf("glusterfs: failed to get endpoints %s[%v]", epName, err) return nil, err </ins> } <ins> glog.V(1).Infof("glusterfs: endpoints %v", ep) </ins> return plugin.newMounterInternal(spec, ep, pod, plugin.host.GetMounter(), exec.New()) }

several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.


Question: ginkgo.It("should verify ResourceQuota's priority class scope (quota set to pod count: 1) against a pod with same priority class.", func() { _, err := f.ClientSet.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: "pclass1"}, Value: int32(1000)}, metav1.CreateOptions{}) <del> framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) </del> <ins> if err != nil && !apierrors.IsAlreadyExists(err) { framework.Failf("unexpected error while creating priority class: %v", err) } </ins> hard := v1.ResourceList{} hard[v1.ResourcePods] = resource.MustParse("1")

The tests are full of assertions which compare a boolean against true or false, often without any additional explanation. When those assertions fail, the error message is useless for understanding what went wrong, often just saying: Including an explanation is better, but then we still have the distracting noise about the comparison: All of those assertions should be replace with . Such locations can be found by grepping: UPDATE (2024-08-26): in cases where such an if check is not possible (typically in combination with or ), should be used instead of . They emit only the explanation given to them, without including the noise above. The Ginkgo linter (enabled in golangci-lint) points that out. Some progress has been made in the meantime: v1.23.0: 250 v1.24.0-beta.0: 177 However, we were down to 175 at some point, so the number is going back up. Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: It makes sense to split this work into individual PRs, one per OWNERS file under test/e2e. Here's a list of directories: [ ] test/e2e/apimachinery [x] test/e2e/apps: [x] test/e2e/auth [x] test/e2e/autoscaling: [x] test/e2e/cloud/gcp [x] test/e2e/cloud [x] test/e2e/common/network [x] test/e2e/common/node: [x] test/e2e/common/storage: [x] test/e2e/framework: [x] test/e2e/framework/providers/vsphere [x] test/e2e/framework/volume [x] test/e2e/instrumentation/logging [x] test/e2e/instrumentation: [x] test/e2e/kubectl [x] test/e2e/lifecycle [x] test/e2e/node: [x] test/e2e/scheduling: [x] test/e2e/storage: (prior PR: ) [x] test/e2e/testing-manifests/storage-csi [x] test/e2e/upgrades/apps: [x] test/e2e/windows: , When creating a PR, use as title. Use one commit per PR, squash when addressing feedback.

Unless noted otherwise, will work on PRs. Help may be welcome. /sig testing /help Better output for developers when tests fail.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: /sig testing /help </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/triage accepted
I would like to take this up as my first issue
/assign
I would like to take up this issue as my first issue
Hello I am already working on this issue , can you please take some other issue ?
Okay sure ,how do I un assign myself from this issue?
I don't think you are assigned to this issue.
Oh okay
Hey as discussed in pr I am assuming that you are only working for tests under area.

also finished the work on , see
and only one PR is to be made for the correction under test/e2e/storage/, so how can we distribute the task and also make only single PR. Or can we make multiple PR's
I have a TODO list to the issue description. I'll try to keep that up-to-date with what has been done and who is working on it.
you can continue with in
what is to be changed in this case !
can you inform on which directory you are working.
Currently working on apimachinery
Nothing, just leave it as-is.
But that isn't under - before starting somewhere else, please coordinate with
test/e2e/instrumentation/logging don't have any cases like "Expect.Equal([^,], *(true|false)", so I think it should be removed from tasks to do.
I am thinking that the output should be if !ok { framework.Failf(uexec.ExitError) } in this case, am I thinking reasonable here.
isn't valid Go. The first parameter must be a format string. The code checks whether err is a : Therefore the change should be:
Why don't we just add extra explanation into calls like for easy investigation? I might miss some point.
You have to consider the complete output when that case fails. Gomega will write print the actual value and the expected value, then in addition you get the same information in the custom message. Printing just the custom failure message is shorter and more readable, in particular for a comparison of booleans. For a comparison of integers I see two solutions: ExpectEqual with a fixed message that just explains what is being compared. An if check with Failf where the message includes the actual value. The later seems more useful when the expected value is zero because that value doesn't need to be printed when its obvious.
I am getting difficulty in figuring out what should be output here, what should the appropriate output here. Also what steps you took in knowing so, it would help to improve my understanding.
The "ok" value is the result of checking for a specific key in a map. What a developer therefore wants to know when that check fails is: what was the key what's in the map Both can be included in:
can I work for the section :- test/e2e/scheduling ?

And there was only one changes needed for test/e2e/node which I did but my PR was closed as it only contains trivial edits. That was my first PR, so can you please guide me further?
/assign I'll try to coordinate the work, but won't create PRs myself. I ticked of line items which had no remaining work. feel free to start working on .
I replied there and also suggested an improvement. For those following this issue: try to include more information than before in the error message when the assertion fails. Then it is more obvious to the reviewer that the change is useful without having to read this issue. Examples:
Another tip: don't just reference this issue as explanation for the PR. Reviewers are busy, they won't necessarily invest the time and follow the link to understand what the PR is about. Instead, take the first sentence from this issue and adapt it as needed to the specific changes in the code.
can I start working on test/e2e/common/storage ?
Okay.
for test/e2e/network/netpol , there are already message associated with all of the cases, for example :- So should I just change all of them to ? And same for all the cases in test/e2e/network except these two cases : framework.ExpectEqual(netutils.I sIPv4String(internalIPs[0]) != netutils.IsIPv4String(internalIPs[1]), true) framework.ExpectEqual(netutils.IsIPv4String(p.Status.PodIPs[0].IP) != netutils.IsIPv4String(p.Status.PodIPs[1].IP), true )
Yes.
Okay then I will be working on and
It might make sense to combine them in a single PR, depending on how many changes there are - just use common sense.
IMO : this will never fail because whether err will be nil or it will exist. So, we don't need to change this code. What I had done in this case was :
Note that that is redundant (= not wrong, just not needed). I later clarified that in
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Hi! I've created a PR to address the tests in e2e/apps It is my first PR so hopefully, everything is ok. Thanks a lot!
Hi can I take test/e2e/common/node?
Yes, please do.

Have we considered / helpers that can skip the comparison output? - is noisy and makes it harder to skim tests, especially given that it's a pretty commonly required pattern.
No, we haven't. Do you find it noisy because it's two lines instead of one? I don't have a problem with that. It seems to be a common approach in unit tests and I find it easy to read, partly because it matches how production code would be written.
It becomes three, plus extra indention - but more importantly stands out in a series of expectations. This draws extra scrutiny to those lines. many unit tests using testify would use to express this because of that.
Isn't that a good thing? is bad without any or no useful extra explanation. The if check makes it obvious that a good failure message is needed. Does print less useless boilerplate code than Gomega? We could use that with Ginkgo (), but then we still need to ensure that the optional explanation is useful.
xref
Hi can I take test/e2e/auth?
for test/e2e/auth, for example : Should I just change it to ?
feel free to work on test/e2e/auth. This one particular example is a bit different: here we are comparing string values, and for those it makes sense to print the actual value because the failure message then adds information. I would leave that one as it is.
OK. test/e2e/auth doesn't need replacing. I : <img width="731" alt="Screen Shot 2022-05-30 at 9 12 59 PM" src="https://user-"There are a lot of positions that need to be modified in . I'll take test/e2e/apps.
That looks like a good candidate. Please go for it. When you need to make up new failure messages, try to think about what information may be useful to have and add that. Then the resulting PR is clearly better than the status quo.
Hi can you please assign me a directory. I want to give it a try. Thanks,
Hi I already posted a PR for e2e/apps some days ago in this issue: If it is fine I'd love to proceed with that. Just to avoid us repeating work.
sorry for not noticing the overlap sooner. Let's go with the older PR.

you can take over test/e2e/storage from There is an incomplete PR for it in You can check out that branch, fix it and submit a new PR, or start from scratch.
it looks like is not active anymore. I've remove him from the todo list and you instead for test/e2e/common/storage.
OK. Thank you. Is there an incomplete PR for .
I in and it returns 0. What should I do?
needs modifying. <img width="1440" alt="Screen Shot 2022-05-31 at 2 40 29 PM" src="https://user-"Can I work on it?
Indeed, did test/e2e/common/storage in You can work on test/e2e/windows.
OK. I'm working on it currently.
Sure, I will look into it. Thanks,
: do you want to continue with test/e2e/apimachinery?
Yes. There's an older PR:
Hi is there anything I could take over?
Hey I am not sure how updated the list is, but I am available to help. Is there any part I could start or continue?
sorry for the late reply. I had (and still have) some big changes for the E2E tests going and had put this on the backseat for a while. If you are interested, how about doing test/e2e/apimachinery and test/e2e/autoscaling? Some work was done on the latter, but there are still several calls (again?).
Sure, I can take that!
Hey I'd also like to contribute on this issue. Is there anything I could take over?
feel free to take my part:
Perfect, then I will take part
Hi :smile: , I would like to contribute to this issue, there is anything that I could do?
I was reluctant to start new work on this because I had one big PR pending that touched a lot of test code (PR ), but that has been merged. you are taking test/e2e/autoscaling, right? test/e2e/apimachinery? you can do test/e2e/apps. That leave test/e2e_node in case someone wants to continue.
yes e2e/autoscaling. I had some trouble to get the tests going initially, but seems fine now!
Hi, Can I do something to contribute to this issue?

you could work on test/e2e_node.
Ok, I will start work on test/e2e_node thanks!
Hi , I would like to contribute. Is there anything I could take over?
thanks for your interest, but right now everything seems to be covered.
Hi Thanks. I am searching for "good-first-issue", feel free to assign one to me. I am exploring api-machinery, but happy to contribute in other sig as well.
maybe you can take a look at if you have some time :)
Hi , There is anything that I could do to contribute with this issue?
Hi , I was looking and found the test/e2e/apimachinery without doing it and only then did I see that it was already assigned if this is a problem just refuse my pr and if you have something else to do let me know.
The remaining work will probably finished in , so there isn't anything else to do. If someone wants to investigate writing a linter for this, then this would be a cool science project - not easy, though. I would start with looking at for this. It has the advantage that it is supported indirectly in golangci-lint ().
Hello I would like to try to do a liter for it. It will be a cool project whit a lot of opportunities to learn. There will be a separate issue for the linter or we continue the discussion here?
Let's continue here.
I ended up creating a separate issue for linter ideas: let's continue there.
Ok!
New tests have been again with this anti-pattern since the last time that I checked. It's clear that without a linter, we'll not get rid of it. So let's focus on that first.
is this issue still open
Yes, this is still open. Current status as of v1.28.0-rc.0: Two calls are in , the rest are in . We have such a linter now, with ginkgolinter in golangci-lint and (soon) the ability to prevent usage of , so it makes sense to continue with removing those calls. do you want to prepare one PR for all remaining calls? I think one PR is enough, no need to split it up.
also has it, which was intentional. I'm undecided whether that should be replaced.

Several of those calls are being touched in , let's wait for that PR to be merged before doing any further work.
no longer finds anything. We might be done... :tada: But not quite. What we have left now is checking a boolean via or , which has the same problem. In some cases, there is an extra message (which is still noisy, see ), in other cases there isn't even that (which makes the failure message fairly useless). We can and should replace some of those calls with the pattern.
is almost resolved. My work-in-progress branch forbids the usage of in favor of the new or plain Go. In some cases, moving the Go code that returns true/false into a matcher might be the best solution because that allows dumping the full actual value that was being checked. Here are the remaining issues: <details</detailsDoes anybody want to work on those?
I would love to!
: please go ahead. Assign PRs to me when you create them. Let's start with a few of those calls initially, do a round of reviews, then continue. /assign
You will have to start with my branch, current master doesn't have gomega with thew new API yet.
I'm starting new to kubernetes. Is there anything easy that I could do ?
Hey Is there anything left I can help you with in this issue ?
Thanks for your interest. There's not much to be done at the moment, though.
Should this be closed?
Oh wow! Looks like we are indeed done. Thanks everyone. /close
Closing this issue. <detailsIn response to : Oh wow! Looks like we are indeed done. Thanks everyone. </details


Question: // according to proxier.ipsetList information and the ipset match relationship that `ipsetWithIptablesChain` specified. // some ipset(kubeClusterIPSet for example) have particular match rules and iptables jump relation should be sync separately. func (proxier *Proxier) writeIptablesRules() { <ins> // Dismiss connects to localhost early in the service chain loAddr := "127.0.0.0/8" if proxier.ipFamily == v1.IPv6Protocol { loAddr = "::1/128" } proxier.natRules.Write("-A", string(kubeServicesChain), "-s", loAddr, "-j", "RETURN") </ins> // We are creating those slices ones here to avoid memory reallocations // in every loop. Note that reuse the memory, instead of doing: //   slice = <some new slice>

Loopback addresses (127.0.0.0/8, ::1/128) can't be used by services in proxy-mode=ipvs it makes sense to dismiss those early, add RETURN rule to KUBE-NODE-PORT,KUBE-SERVICES No response <details</details25 <details</detailsnone <details</details<details</details<details</details<details</details
/sig network
As mentioned in , this is not a bug since the nodeport and ephemeral ranges overlap.
/kind cleanup /kind remove-bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /kind cleanup /kind remove-bug </details
/remove-kind bug
The title of this issue should be changed to something like; Which would indicate that this is not a bug fix but an improvement. Also remove the invalid use-case in the description and add references to comments in like I think to dismiss nodePort connects to localhost early for proxy-mode=ipvs is an improvement, and in the majority of systems traffic to localhost is more common than traffic to nodePorts so this change will also improve overall performance. This is not a fix for overlapping ephemeral and nodePort ranges! So if the title and description is not updated the correct triage is to close this issue as it violates the non overlapping rule for ephemeral/nodeport ranges, please see
/area ipvs
Ok, i will change issue title and descripiton
/triage accepted


Question: <ins> # 3.8 * Updating base image for Windows container images from nanoserver to `mcr.microsoft.com/oss/kubernetes/windows-pause-image-base` which gets built on a Windows machine. This addresses [Cannot modify registry keys during Windows pause image build process #109161](https://github.com/kubernetes/kubernetes/issues/109161) </ins> # 3.7 * Unsupported Windows Semi-Annual container images removed (OS Versions removed: 1903, 1909, 2004) . ([#107056](https://github.com/kubernetes/kubernetes/pull/107056), [@jsturtevant](https://github.com/jsturtevant/))

The Windows pause images is built using buildkit on Linux hosts and this configuration does not allow for the Windows registry to me modified during the container build process. When Windows containers are built on Windows host machines part of the build process creates 'delta' registry files when get stored in the layer files. These 'delta' registry hive files are applied to the containers active registry when the containers are created. Chaning the registry files in the container image layers without having the 'delta' files present will result in the changes NOT being present in the container image. We have an increasing demand for the ability to modify registry in the pause image on Windows such as Disable some services like DiagTrack that negatively impact performance Change behavior of RPCSS so it doesn't hijack RPC port (135) for all containers in the pod Things are working as expected given the current build process. There are linux utilities like that can modify Windows registry files that I have used to test this out. After running the container produced from these files you can see the registry changes made by persisted in the container image but these changes are not present in the container's active registry. vs One possible solution is to create an intermediate image based on (the current base image for the Windows pause image) that performs all the desired registry operations that is built on a Windows machine and published someplace (MCR maybe?) and the use that as the container base image for the Windows pause image. <detailsN/A </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
cc
my first concern was. will a community member be able to build this image themselves (will the sources be available and up-to-date and the image built from it). We could also do a github hosted windows runner to build / publish this new image as well. let's please revisit this once we wrap 1.24.

/milestone v1.25
We created which has the source/docker files used to build . AKS has made a pause image based on image available to some customers out of necessity to unblock some scenarios. We'd like to upstream the changes - and do so with as little destabilization to Kubernetes project as possible.
Seems like the intermediate base image step would be reasonable. I don't think community members can build the existing base windows images either for the same reason of depending on windows?
The windows pause image is currently based on 'nanoserver' which is one of several container base images provided by Microsoft (and only Microsoft can produce container base images).
yep. i'd expect community downstream teams (for example) would like to start with this nanoserver, have a set of dockerfiles to build this new image ... then build their own pause images based on this new image
or in other words if MSFT wants to roll all the changes into nanoserver itself (since we all already know that only MSFT can produce this), then there is no debate after that
We can't roll these changes into nanoserver because they may break other scenarios. Because of that - having an intermediate layer built somewhere was the best solution we came up with without forking the build process for the pause image. Microsoft is willing to host build/host the intermediate layers/images from open-sourced docker files. If we want to build the intermediate layers somewhere else too (like github hosted Windows runners) that would be fine too.
/sig windows
If this intermediate image used only by k8s then it is a good idea to build the image via GH actions in the kubernetes org so that k8s maintainers have better control on this or if this image is going to be generic for anyone to use then MSFT can build one and host it somewhere.
Picking this back up since 1.24 release is behind everyone. - Do you think a separate github repo for the dockerfile and github actions staging/prod container repos would be appropriate here? If so I can get started on creating these. Also - Is it possible to push to the staging repos from a github action job instead of PROW jobs? We'd need to use github actions to build the Windows image on a Windows machine.

I supposed we can store the .tar files as release artifacts if needed and then use a PROW to job push them to the staging repo if needed but being able to push for the action would be simpler.
/cc as FYI
I see two possible paths forward Continue to host the source for the intermediate layer at , publish the image to , and then update the Windows pause build docker files in k/k to consume this image instead of the or Create a new kubernetes-sigs repo to host the source and build steps for the image, publish the image to , and then update the Windows pause build docker files in k/k to consume this image. I'm fine with either approach but am not sure who the right folks to discuss this with are.
This looks more promising because this image you are creating is not a specific change for the pause container, IMO if we can change that image name to something generic and use the same wherever required.
I think those both sound reasonable personally. I think it's technically feasible to push from GitHub actions to staging GCR with workload identity but nobody has done this yet in Kubernetes.


Question: if ns != "" { nsArg = fmt.Sprintf("--namespace=%s", ns) } <del> for _, selector := range selectors { resources := RunKubectlOrDie("get", "rc,svc", "-l", selector, "--no-headers", nsArg) if resources != "" { Failf("Resources left running after stop:n%s", resources) } pods := RunKubectlOrDie("get", "pods", "-l", selector, nsArg, "-o", "go-template={{ range .items }}{{ if not .metadata.deletionTimestamp }}{{ .metadata.name }}{{ "n" }}{{ end }}{{ end }}") if pods != "" { Failf("Pods left unterminated after stop:n%s", pods) </del> <ins> backoff := wait.Backoff{ Duration: 5 * time.Second, Factor:   2, Steps:    3, } var e error verifyCleanupFunc := func() (bool, error) { e = nil for _, selector := range selectors { resources := RunKubectlOrDie("get", "rc,svc", "-l", selector, "--no-headers", nsArg) if resources != "" { e = fmt.Errorf("Resources left running after stop:n%s", resources) return false, nil } pods := RunKubectlOrDie("get", "pods", "-l", selector, nsArg, "-o", "go-template={{ range .items }}{{ if not .metadata.deletionTimestamp }}{{ .metadata.name }}{{ "n" }}{{ end }}{{ end }}") if pods != "" { e = fmt.Errorf("Pods left unterminated after stop:n%s", pods) return false, nil } </ins> } <ins> return true, nil } err := wait.ExponentialBackoff(backoff, verifyCleanupFunc) if err != nil { Failf(e.Error()) </ins> } }

/priority failing-test /priority important-soon /sig cli /kind flake
Reiterating the mentions to trigger a notification: <detailsIn response to : ### Flaky Job: pr:pull-kubernetes-e2e-gce /priority important-soon /sig cli /kind flake Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here's the TestGrid view to dig into the flakes and get more detailed logs https://k8s-
/assign
GitHub didn't allow me to assign the following users: Liujingfang1. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The failure is that after the replication controller is cleaned up, the pods are not terminated. I investigated the failure. It is from function after . does two things: make sure the resource is deleted and make sure the pods are deleted. If the pods haven't started termination during the running, we will see this failure. The solution is to add several retry, give the pods more time to start termination.
Liujingfang1@ I see you have a PR out for the fix, thanks. Any ETA on when this will be merged? I can monitor test status once its in before pinging you to close the issue.
ETA is this week. Maybe tomorrow :)
can we plz reopen this issue and close it once we see the flake is actually fixed? Thanks
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
what's wrong with having it closed and re-open if the flake hits back?
there is no harm in doing it either way. The only reason I prefer keeping it open is because it filters in my customized query tracking all open issues created by me. Its easier to verify the fix that way versus going after .

closed issues that fell off the list.
sgtm as long as you'll don't forget to close it when confirmed :wink: The reason I usually close them is so that I don't have to remember.
I will definitely keep an eye out and close it :) I usually check on these before my weekly flake report.
I do see a few flakes with this test still https://k8s- Both failures are the same: Apr 2 21:35:55.396: Pods left unterminated after stop: update-demo-nautilus-dggjz update-demo-nautilus-zb8pv Gubernator results: https://k8s- https://k8s-
I've opened and increased the timeout to 1 minute similarly to the time GC tests are waiting for cleanups. ptal
it looks like the last linked PR fixed the problem and from looking at the testgrid it hasn't appeared, mind closing?
Thanks for the fix. The test seems to pass fine in the GCE job. However I do see a couple fo flakes in the GKE job. https://k8s- https://k8s- Let me know if this needs a different fix/investigation. If so I am happy to close this and open a new issue.
There are only 2 test failure happened from 03/27 to 04/10. And these 2 failures are high likely to be related to some cross-cutting infra failure, e.g. https://k8s- I believe we can close this issue.
Sounds good. I am closing this issue now. Will reopen if it surfaces as top flake again. Thanks for your efforts here.
/close


Question: } if e, ok := out.(runtime.NestedObjectEncoder); ok { <del> if err := e.EncodeNestedObjects(DirectEncoder{Encoder: c.encoder, ObjectTyper: c.typer}); err != nil { </del> <ins> if err := e.EncodeNestedObjects(DirectEncoder{Version: c.encodeVersion, Encoder: c.encoder, ObjectTyper: c.typer}); err != nil { </ins> return err } }

When an object is told to encode its nested objects, the preferred encoding version is not provided for it to use. That means nested objects with more than one possible serialized version use the default instead of the requested version.


Question: cmdutil.CheckErr(o.Run()) }, } <del> cmd.Flags().BoolVar(&o.Recursive, "recursive", o.Recursive, "Print the fields of fields (Currently only 1 level deep)") cmd.Flags().StringVar(&o.APIVersion, "api-version", o.APIVersion, "Get different explanations for particular API version (API group/version)") </del> <ins> cmd.Flags().BoolVar(&o.Recursive, "recursive", o.Recursive, "When true, print the name of all the fields recursively. Otherwise, print the available fields with their description.") cmd.Flags().StringVar(&o.APIVersion, "api-version", o.APIVersion, "Use given api-version (group/version) of the resource.") </ins> // Only enable --output as a valid flag if the feature is enabled <del> cmd.Flags().StringVar(&o.OutputFormat, "output", plaintextTemplateName, "Format in which to render the schema (plaintext, plaintext-openapiv2)") </del> <ins> cmd.Flags().StringVar(&o.OutputFormat, "output", plaintextTemplateName, "Format in which to render the schema. Valid values are: (plaintext, plaintext-openapiv2).") </ins> return cmd }

Especially the options description are super sad. I think we can improve them in the next beta iteration cc /sig cli
/triage accepted
Hi does fixes this issue?
I agree with the same. Unlike other outputs, it doesn't share the details about the values or their usage.
Unfortunately, I don't think it does. Have you tried it? Please send a pull-request I'll happily review it.
Hi , I'd like to work on this. However, I would need some guidance so I know I'm in the right direction. My guess is we change the text I have noticed these descriptions are not as per the other flag descriptions in kubectl. I'll make a draft once you confirm.
For eg these are options in : we can try adding details and formal tone to the text. /assign
I don't know if you need my help, you seem to be doing just fine. Please go ahead and ask me questions if you have specific problems, I'll be happy to guide. Thanks!
That's correct.
sure thing , I'll send the PR
Hi How does this output looks :
Hi Raised the PR: for reviews on same.


Question: StartProcess-WriteSshKeys } <del> Set-EnvironmentVars </del> Create-Directories Download-HelperScripts

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Docker is always installed on node boot which adds significant start time and adds software that's unnecessary. containerd Windows nodes should not have Docker installed. Start a Kubernetes GCE Windows cluster via kube-up. You can run stuff like on it. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig windows /triage accepted


Question: t.Errorf("unexpected objects differerence after modifying status: %v", diff.ObjectDiff(statusVolumeAttachment, expectedVolumeAttachment)) } <del> // modifying spec should be dropped </del> <ins> // spec and metadata modifications should be dropped </ins> newVolumeAttachment := volumeAttachment.DeepCopy() newVolumeAttachment.Spec.NodeName = "valid-node-2" <ins> newVolumeAttachment.Labels = map[string]string{"foo": "bar"} newVolumeAttachment.Annotations = map[string]string{"foo": "baz"} newVolumeAttachment.OwnerReferences = []metav1.OwnerReference{ { APIVersion: "v1", Kind:       "Pod", Name:       "Foo", }, } </ins> StatusStrategy.PrepareForUpdate(ctx, newVolumeAttachment, volumeAttachment) if !apiequality.Semantic.DeepEqual(newVolumeAttachment, volumeAttachment) {

This came up as a follow up to Also rather than opening 5 different issues - we are going to use this github issue to track all follow up items mentioned here -
/sig storage
/assign /assign
: PR
and the rest is fixed in


Question: "fmt" "path" "strings" <ins> "sync" </ins> "time" "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We found that cloud provider tries to attach a disk, it totally costs 121s in the end one minute(disk still in being detaching state), azure cloud provider tries to delete that disk and it succeeded 121s, attach disk operation succeeded and finally azure vm has attached one non-existing disk What you expected to happen: should not delete an azure disk when that disk is being attached How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: The difficulty here is when a disk is being attached or detached, the disk state is still "unattached", would be the reason why delete disk at that time would succeed: Azure disk RP would have two options to fix this issue: a new disk state: , disk delete operation when disk is being attached or detached Otherwise on azure provider(client) side, we cannot know whether that disk is being attached or detached since the second delete disk operation is done after 1 min Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: echo "Starting etcd" ETCD_DIR=$(mktemp -d -t kube-integration.XXXXXX) <del> trap "rm -rf ${ETCD_DIR}" EXIT </del> (etcd -name test -data-dir ${ETCD_DIR} &> /tmp/etcd.log) & ETCD_PID=$!

For structured logging migration we have developed a tool to klog invocations For a given directory logcheck validates two things: directory is fully migrated - check if there are non-structured logging calls like usage of and is correct - check if key values passed are paired. We run logcheck as part of Kubernetes verify scripts on directories listed in Directories listed in are marked as migrated to avoid regressions from organized migration, however there are also other code locations that already use new functions that are not yet ready to be marked as migrated (components that are not part of organized migrations are done as best effort). We should split checking if directory was migrated from correctness of new functions and scan whole codebase for correctness. Plan: Add a flag to logcheck for requiring that directory was migrated. Add to verify-strucured- running logcheck of whole k/k codebase to check correctness This will ensure higher quality of migration PRs by validating correctness of directories that were not yet in .structuredlogging /wg structured-logging /cc
I'd like to pick this up. Do I need to wait till it is triage/accepted?
/triage accepted
Feel free to pick it up :P Usually I don't want to self triage, but this issue is purely Structured Logging WG related.
/assign
[Case 2: Correct usage] If the directory is not marked as migrated, we check that it's usage of is correct and don't care about usage of . By default, CI will test for correctness of whole k/k codebase instead of testing for complete migration (which is the current behavior). And to test for migration, a new flag must be passed explicitly. is this the expectation?
To clarify, let me rephrase. Current state is What we want: [Case 2: Correct usage] tested on whole repository
If this is the case, I can start implementing it.
SG, thanks for the help.
/milestone v1.22 Part of work go graduate Structured Logging KEP to beta.
I don't think work here has finished. We still need to implement those changes in k/k repo: Cut new klog Bump klog in k/k Update the script ./hack/verify-structured- /reopen
Reopened this issue. <detailsIn response to : I don't think work here has finished.

We still need to implement those changes in k/k repo: Update the script /reopen </details
Why do we need to cut a new klog and bump in k/k? Can't we make latest logcheck available to k/k without it? I don't know the process.
Hmm, We could try without updating to update. Please send PR that will update ./hack/verify-structured- If updating klog is nesesery feel free to contact for help
If we do both, we'd have to run logcheck twice (with and without flag) which means test time will increase.
This depends on implementation, increasing running time is ok as long it's reasonable, for example x1.5, but not x10.
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
We would like to have static analysis setup before we start migrating logs in 1.23. will you be able to work on it this week? Do you need any help?
Yes, I can. PR kubernetes/kubernetes/pull/ is up for enabling static analysis. It has some unresolved conversations. It'd be helpful if I could get another round of review on that PR. Before we merge that we'd need to fix existing logging issues which are listed here: If I could get help fixing those it'd be great. Else I can fix it too.
I see two issues that need to be addressed: Fix handling of in klog code, this should be done by creating an issue to klog () Fix remaining errors Upgrade klog version to pick up fix for Exit


Question: cp "${KUBE_SERVER_BINARIES[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/server/bin/" <ins> # Include the client binaries here too as they are useful debugging tools. cp "${KUBE_CLIENT_BINARIES[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/server/bin/" </ins> local package_name="${RELEASE_DIR}/kubernetes-server-${platform_tag}.tar.gz" kube::release::create_tarball "${package_name}" "${release_stage}/.." done

Currently the master VM specified by the Vagrantfile does not have the built kubecfg binary in the vagrant user's path. This request is to have this binary present on the host system and in the vagrant user's path so I can Vagrant ssh into the machine and do 'kubecfg list pods'.
I'm going to generalize this a little so that kubecfg is available in our general cluster install scripts. This should be easier as we move to binary installs.
Should be kubectl, not kubecfg, but otherwise is a good idea. etcd tools would be useful, too:
I'll take this on if no one else wants to do it. May take me a day or two to get to it.
Thinking out loud here... do we really want to encourage people to do admin and/or job runny stuff on their master?
I see it as a debugging tool.


Question: path: ../_output/local/bin/logcheck.so description: structured logging checker original-url: k8s.io/logtools/logcheck <ins> forbidigo: analyze-types: true forbid: - p: ^managedfields.ExtractInto$ pkg: ^k8s.io/apimachinery/pkg/util/managedfields$ msg: should not be used because managedFields was removed - p: .Extract pkg: ^k8s.io/client-go/applyconfigurations/ msg: should not be used because managedFields was removed </ins> gocritic: staticcheck: checks:

Add a code static check for to report using server-side-apply extract With optimization of and , the will be removed in and , and maybe more of other components in the future, using Extract is problematic. It's safer to add this lint check.
/cc
/sig api-machinery
/sig scheduling /sig apps
/assign /triage accepted


Question: } w.Write(LEVEL_1, "%stn", host) for _, path := range rules.HTTP.Paths { <del> w.Write(LEVEL_2, "t%s t%s (%s)n", path.Path, backendStringer(&path.Backend), i.describeBackend(ns, &path.Backend)) </del> <ins> w.Write(LEVEL_2, "t%s t%s (%s)n", path.Path, backendStringer(&path.Backend), i.describeBackend(ing.Namespace, &path.Backend)) </ins> } } if count == 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The issue is with the information provided for the rules, not about the default backend. Form the describe: Rules: Host Path Backends ---- ---- -------- hellonode hellonode:8080 (<none) the endpoint show none. and below is the ingress yaml file. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hellonode spec: rules: host: hellonode http: paths: backend: serviceName: hellonode servicePort: 8080 What you expected to happen: it should show the service endpoint How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/area kubectl
/sig cli


Question: } } <ins> func TestNodeAddressUpdatesGenerateAPIPodStatusHostNetworkPodIPs(t *testing.T) { testcases := []struct { name           string nodeIPs        []string nodeAddresses  []v1.NodeAddress expectedPodIPs []v1.PodIP }{ { name:    "Immutable after update node addresses single-stack", nodeIPs: []string{"10.0.0.1"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "1.1.1.1"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:    "Immutable after update node addresses dual-stack - primary address", nodeIPs: []string{"10.0.0.1", "2001:db8::2"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "1.1.1.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, { name:    "Immutable after update node addresses dual-stack - secondary address", nodeIPs: []string{"10.0.0.1", "2001:db8::2"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.0.0.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8:1:2:3::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, { name:    "Immutable after update node addresses dual-stack - primary and secondary address", nodeIPs: []string{"10.0.0.1", "2001:db8::2"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "1.1.1.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8:1:2:3::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, { name:    "Update secondary after new secondary address dual-stack", nodeIPs: []string{"10.0.0.1"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.0.0.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kl := testKubelet.kubelet for _, ip := range tc.nodeIPs { kl.nodeIPs = append(kl.nodeIPs, netutils.ParseIPSloppy(ip)) } kl.nodeLister = testNodeLister{nodes: []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{Name: string(kl.nodeName)}, Status: v1.NodeStatus{ Addresses: tc.nodeAddresses, }, }, }} pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") pod.Spec.HostNetwork = true for _, ip := range tc.nodeIPs { pod.Status.PodIPs = append(pod.Status.PodIPs, v1.PodIP{IP: ip}) } if len(pod.Status.PodIPs) > 0 { pod.Status.PodIP = pod.Status.PodIPs[0].IP } // set old status podStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, } podStatus.IPs = tc.nodeIPs status := kl.generateAPIPodStatus(pod, podStatus) if !reflect.DeepEqual(status.PodIPs, tc.expectedPodIPs) { t.Fatalf("Expected PodIPs %#v, got %#v", tc.expectedPodIPs, status.PodIPs) } if kl.nodeIPs[0].String() != status.PodIPs[0].IP { t.Fatalf("Expected HostIP %q to equal PodIPs[0].IP %q", status.HostIP, status.PodIPs[0].IP) } }) } } </ins> func TestGenerateAPIPodStatusPodIPs(t *testing.T) { testcases := []struct { name      string

What happened: We have installed k8s cluster on openstack cloud provider setup. K8s cluster is dual-stack enabled. During fresh deployment or worker node reboots (master nodes , worker nodes) , k8s control plane pods (other host networking pods aswell) such as kube-proxy, kube-scheduler , kube-controller-manager get only 1 IP address instead of dual-stack address. What you expected to happen: We expect all host networking pods to get dual-stack addresses , since cluster is dual-stack enabled. How to reproduce it (as minimally and precisely as possible): a) Openstack cloud provider setup with minimum 4-5 VMs. b) 3 master nodes, 2 worker nodes c) Execute simultaneous reboot of all nodes. Environment: Kubernetes version : v1.22.0
/sig node
/sig network /triage needs-information /assign Hi When you describe your node what are the IPs assigned in node object? kubelet copies from node object into pods. The way they are set on node object is by the cloud provider, manual via kubelet's flag or auto-detect which prefers IPv4 (check: ).
when i describe the node object , they contain dual-stack addresses. Yes i debugged kubelet with few additional logs. I could see that when kubelet is reading node obj to create these static pods, it sees only 1 IP. But eventually node object is populated with both the addresses. I also understand, is for baremetal installation. But we have cloud provider setup which might be delaying setting both the IPs in node obj. Before cloud provider sets both the IPs ,kubelet is fast enough to create these static pods. Let me know if you need any additional logs.
thanks again Few additional questions: are these static pods? When the node object gets its dualstack addresses and pods were created after the node object updates do you still get single address pods?
I've chatted with the reporter in slack, it seems a race on populating the node addresses by the cloud-provider,
Yes. It would seem so. Two things here i am trying to clarify. cloud providers node register with taints (no schedule) until cloud provider adds provider-Id and other details including IPs.

these are static pods then I am not sure if kubelet respects no-schedule (those are respected by scheduler which is out of the picture for static pods) If they are then they will run - most likely - before node gets IP addresses from cloud provider. The questions i asked above should put some lights on this.
Yes i am referring to kube-proxy , kube-api-server ,kube-scheduler pods which use host-networking. This is not under our control. I mean , we are just rebooting the nodes in quick succession and at the end we just check the IP addresses of the above static pods. As i see the code , kubelet brings them up as soon as he is UP.
IMO the cloud provider really needs to figure out the IPs before kubelet starts. OR the node startup needs to block until both IP addresses are available so that the cloud provider and/or kubelet can read them.
Should we accept this as a bug? Do we think we need to force some taint on new nodes that cloud-providers need to clear (obviously not compatible, but you get the idea)?
-- There is a no-schedule taint already there (if cloud provider is configured on kubelet). It is a bug (or undocumented behavior). I think we should always update PodIPs in host network mode. /triage accepted
xref
so i looked at this again. I believe should fix this issue as well. i don't think we should place this behind a feature gate. We already update when the host IP changes. We just need to edit the release note of to describe the behavior and caveats. Thoughts?
Please ignore that comment. The PR does not solve for
/remove-sig node
: Any more info needed from me ?
no thank you. We are addressing this by and the related KEP.
, any idea when will the fix be available ?