Question: return res, nil } <ins> func (h *FakeNetlinkHandle) GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) { res := sets.New[string]() for linkName := range h.localAddresses { if linkName == dev { continue } for _, addr := range h.localAddresses[linkName] { if h.isValidForSet(addr) { res.Insert(addr) } } } return res, nil } </ins> // SetLocalAddresses set IP addresses to the given interface device.  It's not part of interface. func (h *FakeNetlinkHandle) SetLocalAddresses(dev string, ips ...string) error { if h.localAddresses == nil {

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: expectInvocations: []invocation{ {path: "/mutating/1/5s", timeoutSeconds: 3}, // from request }, <del> expectError:   true, errorContains: "the server was unable to return a response in the time allotted", </del> <ins> expectError: true, errorContainsAnyOf: []string{ // refer to https://github.com/kubernetes/kubernetes/issues/98606#issuecomment-774832633 // for the reason for triggering this scenario "stream error", "the server was unable to return a response in the time allotted", }, </ins> }, }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: (regardlessoffailurepolicy)and_fail Testgrid link: Reason for failure: and Anything else we need to know: Spyglass: <!-- Please see the deflaking doc () for more guidance! --
/sig api-machinery /area apiserver
Can confirm this PR:
/assign
/triage accepted
The kube-apiserver processing flow is as follows, (non-critical paths are omitted) ... // other filters parse the http client timeout setting, sets the Ctx with Deadline in the http.Request ...// other filters select the http.Request.Context().Done() Pay attention to the processing logic in . Although the Context of http.Request has been cancelled, the Mutex on baseTimeoutWriter needs to be obtained between entering the function. The inner-handler may have written , ready to write , but the current code flow thinks that it should be Panic quickly, and it is recovered by and sends http2.Reset to the Client. As a result, a appears on the client side.
Currently seeing it in periodic and in presubmit for
Regardless of WithCache or WithoutCache, here is not reasonable.
/priority important-soon /milestone v1.21
PS: There were some recent failures:
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-integration per: triage link: FYI
may be one fix, may also be necessary (thanks for opening these
I believe that can effectively repair the issue, but currently is not the key PR to fix the issue.


Question: <ins> load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library") go_library( name = "go_default_library", srcs = ["check_conformance_test_requirements.go"], importpath = "k8s.io/kubernetes/hack/conformance", visibility = ["//visibility:private"], deps = [ "//vendor/github.com/pkg/errors:go_default_library", "//vendor/github.com/spf13/cobra:go_default_library", ], ) go_binary( name = "conformance", embed = [":go_default_library"], visibility = ["//visibility:public"], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], visibility = ["//visibility:public"], ) </ins>

There are a number of conformance test requirements and guidelines: We should try to automate enforcement to the degree we can, via a "linting" tool, and/or test framework checks. For instance, conformance tests shouldn't invoke framework.ProviderIs or framework.SkipUnlessProviderIs. Those could actually check whether they are invoked by a conformance test, and then die. cc /area conformance
/sig testing
/cc


Question: // Initialize CPU manager if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.CPUManager) { <del> cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), podStatusProvider, runtimeService) </del> <ins> cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService) </ins> } // cache the node Info including resource capacity and

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: // Config return a scheduler config object func (o *Options) Config() (*schedulerappconfig.Config, error) { <ins> c := &schedulerappconfig.Config{} if err := o.ApplyTo(c); err != nil { return nil, err } </ins> // prepare kube clients. <del> client, leaderElectionClient, eventClient, err := createClients(o.ComponentConfig.ClientConnection, o.Master) </del> <ins> client, leaderElectionClient, eventClient, err := createClients(c.ComponentConfig.ClientConnection, o.Master) </ins> if err != nil { return nil, err } // Prepare event clients. eventBroadcaster := record.NewBroadcaster() <del> recorder := eventBroadcaster.NewRecorder(legacyscheme.Scheme, corev1.EventSource{Component: o.ComponentConfig.SchedulerName}) </del> <ins> recorder := eventBroadcaster.NewRecorder(legacyscheme.Scheme, corev1.EventSource{Component: c.ComponentConfig.SchedulerName}) </ins> // Set up leader election if enabled. var leaderElectionConfig *leaderelection.LeaderElectionConfig <del> if o.ComponentConfig.LeaderElection.LeaderElect { leaderElectionConfig, err = makeLeaderElectionConfig(o.ComponentConfig.LeaderElection, leaderElectionClient, recorder) </del> <ins> if c.ComponentConfig.LeaderElection.LeaderElect { leaderElectionConfig, err = makeLeaderElectionConfig(c.ComponentConfig.LeaderElection, leaderElectionClient, recorder) </ins> if err != nil { return nil, err } } <del> c := &schedulerappconfig.Config{ Client:          client, InformerFactory: informers.NewSharedInformerFactory(client, 0), PodInformer:     factory.NewPodInformer(client, 0), EventClient:     eventClient, Recorder:        recorder, Broadcaster:     eventBroadcaster, LeaderElection:  leaderElectionConfig, } if err := o.ApplyTo(c); err != nil { return nil, err } </del> <ins> c.Client = client c.InformerFactory = informers.NewSharedInformerFactory(client, 0) c.PodInformer = factory.NewPodInformer(client, 0) c.EventClient = eventClient c.Recorder = recorder c.Broadcaster = eventBroadcaster c.LeaderElection = leaderElectionConfig </ins> return c, nil }

/kind bug /sig scheduling What happened: kube-scheduler that is pointed to a componentconfig file having ignores that it should use this kubeconfig file. Instead, it uses the in-cluster config (as is neither provided). What you expected to happen: The kube-scheduler should respect the kubeconfig provided in the . How to reproduce it (as minimally and precisely as possible): Starting image with whereby the content of looks like this: makes the scheduler logging The exact same configuration as above works with branch. Seems like has introduced the behaviour. The kube clients are created but the componentconfig file is loaded . /cc Environment: Kubernetes version (use ): 1.11.0-rc.2
/milestone v1.11 for triage
Could you please take a look at this? It seems related to the changes you made recently.
/assign
/assign /priority critical-urgent bumping priority since this would be a 1.11 regression
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: allErrs = append(allErrs, field.Forbidden(field.NewPath("spec", "podCIDR"), "node updates may not change podCIDR except from "" to valid")) } } <ins> // Allow controller manager updating provider ID when not set if len(oldNode.Spec.ProviderID) == 0 { oldNode.Spec.ProviderID = node.Spec.ProviderID } else { if oldNode.Spec.ProviderID != node.Spec.ProviderID { allErrs = append(allErrs, field.Forbidden(field.NewPath("spec", "providerID"), "node updates may not change providerID except from "" to valid")) } } </ins> // TODO: move reset function to its own location // Ignore metadata changes now that they have been tested oldNode.ObjectMeta = node.ObjectMeta

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Follow up for , With the change for cloud-controller-manager updating providerID, the is to be updated. But it will fail later in node update call: Should also check validation rule: By adding following line will do the trick. But this will also allow users to manually edit node's providerID Shall we enforce that only cloud-controller-manager can update the providerID? What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cluster-lifecycle
cc
/area cloudprovider
being able to update seems pretty reasonable to me. Can you get a PR in for that since we're close to code freeze?
cc
I found this issue when validating against a cluster running azure cloud provider, could you please also check whether it is the same case from you side? serves somewhat like an identity for nodes, thus it wouldn't expect arbitrarily updating. A compromise proposal is to allow updates when the value is not set. This gives controller manager a chance to set it during node register. Created as a fix proposal.
thanks for the patch, I think I'll have some time this weekend to verify this fix!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/cluster-lifecycle: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details


Question: } } <ins> // createMissingPersistentVolumeClaims creates all of the required PersistentVolumeClaims for pod, and updates its retention policy func (spc *StatefulPodControl) createMissingPersistentVolumeClaims(set *apps.StatefulSet, pod *v1.Pod) error { if err := spc.createPersistentVolumeClaims(set, pod); err != nil { return err } if utilfeature.DefaultFeatureGate.Enabled(features.StatefulSetAutoDeletePVC) { // Set PVC policy as much as is possible at this point. if err := spc.UpdatePodClaimForRetentionPolicy(set, pod); err != nil { spc.recordPodEvent("update", set, pod, err) return err } } return nil } </ins> // createPersistentVolumeClaims creates all of the required PersistentVolumeClaims for pod, which must be a member of // set. If all of the claims for Pod are successfully created, the returned error is nil. If creation fails, this method // may be called again until no error is returned, indicating the PersistentVolumeClaims for pod are consistent with

User cordons a node with a pvc bound to a local persistent volume User deletes a stateful set pod from that node The pod is in pending because the pvc is still bound to a local pv on the cordoned node (expected) The node dies and the pvc is deleted The pod is still stuck in pending because the pvc is missing (bug) The pod now needs to be manually deleted so that the pod+pvc can be recreated After the pvc was deleted, the statefulset controller should have recreated it and bound it to a new node's local pv so that the pending pod could be scheduled. Currently the only way to get the pod out of pending is to manually delete it so that it recreates itself and its pvc We used for the local PVs, but the problem is not limited to this particular provisioner create a stateful set with a pvc that is bound to a local PV use to cordon a node with a local pv delete a statefulset pod from that node delete the pvc from that node the pod is now stuck in pending due to a missing pvc There was a related issue but this only fixed the issue when the order of operations was delete pvc -delete pod. but in this case, it is delete pod -delete pvc <details</details<detailsTested on aws and gcp </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
This issue was fixed by The new issues look different. /close
Closing this issue. <detailsIn response to : This issue was fixed by The new issues look different. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // descending order {buildByImageSize(), 0, 2, false}, {buildByImageSize(), 1, 0, true}, <ins> {buildByImageSize(), 3, 2, true}, </ins> } for _, fooTest := range fooTests {

What happened: When two or more images on a given node have the same size, their order inside the field set by the process is not deterministic. This problem has the side effect of continually triggering controllers which assume this list to be always equal on node updates unless images are or removed. What you expected to happen: The order of images should be stable even when two images have the same size, using the image ID as a second sort field would help this. How to reproduce it (as minimally and precisely as possible): Choose a node and run: Run a few times: It should be possible to see and images alternating on the output. Anything else we need to know?: Our particular use case make it really common for multiple images to have the same size on a given node. I came across this problem while debugging slow updates to DaemonSets and finding out that the kube-controller-manager was leaking memory until it eventually dies due to OOM: <img width="800" alt="Screen Shot 2019-06-30 at 20 27 22" src="https://user-"I was able to track this leak to a large number of watched elements in memory and by taking a look at the logs with increased verbosity I figured it was due to too many Node notifications being processed by the DaemonSet controller: <img width="800" alt="Screen Shot 2019-06-30 at 20 36 23" src="https://user-"It's possible that other controllers in the wild that watch nodes and only ignore the conditions timestamps on updates to face the same problem. I'll send a simple PR to fix this shortly. Environment: Kubernetes version (use ): But by taking a look at the code this problem also happens on master.
/sig node
Nice debugging here :)


Question: fs.StringVar(&o.hostnameOverride, "hostname-override", o.hostnameOverride, "If non-empty, will be used as the name of the Node that kube-proxy is running on. If unset, the node name is assumed to be the same as the node's hostname.") fs.Var(&utilflag.IPVar{Val: &o.config.BindAddress}, "bind-address", "Overrides kube-proxy's idea of what its node's primary IP is. Note that the name is a historical artifact, and kube-proxy does not actually bind any sockets to this IP. This parameter is ignored if a config file is specified by --config.") <del> fs.Var(&utilflag.IPPortVar{Val: &o.config.HealthzBindAddress}, "healthz-bind-address", "The IP address and port for the health check server to serve on, defaulting to "0.0.0.0:10256" (if --bind-address is unset or IPv4), or "[::]:10256" (if --bind-address is IPv6). Set empty to disable. This parameter is ignored if a config file is specified by --config.") fs.Var(&utilflag.IPPortVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address and port for the metrics server to serve on, defaulting to "127.0.0.1:10249" (if --bind-address is unset or IPv4), or "[::1]:10249" (if --bind-address is IPv6). (Set to "0.0.0.0:10249" / "[::]:10249" to bind on all interfaces.) Set empty to disable. This parameter is ignored if a config file is specified by --config.") </del> <ins> fs.Var(&utilflag.IPPortVar{Val: &o.config.HealthzBindAddress}, "healthz-bind-address", "The IP address and port for the health check server to serve on, defaulting to "0.0.0.0:10256". This parameter is ignored if a config file is specified by --config.") fs.Var(&utilflag.IPPortVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address and port for the metrics server to serve on, defaulting to "127.0.0.1:10249". (Set to "0.0.0.0:10249" / "[::]:10249" to bind on all interfaces.) Set empty to disable. This parameter is ignored if a config file is specified by --config.") </ins> fs.BoolVar(&o.config.BindAddressHardFail, "bind-address-hard-fail", o.config.BindAddressHardFail, "If true kube-proxy will treat failure to bind to a port as fatal and exit") fs.BoolVar(&o.config.EnableProfiling, "profiling", o.config.EnableProfiling, "If true enables profiling via web interface on /debug/pprof handler. This parameter is ignored if a config file is specified by --config.") fs.StringVar(&o.config.ShowHiddenMetricsForVersion, "show-hidden-metrics-for-version", o.config.ShowHiddenMetricsForVersion,

The parameter of kube-proxy is configured as ipv6 The metrics port of kube-proxy listens to the ipv4 address  When the parameter of kube-proxy is configured as ipv6, the ip address of metrics listens to ::1 by default The parameter of kube-proxy is configured as ipv6 No response <detailsClient Version: v1.28.3 Kustomize Version: v5.0.4-0.- Server Version: v1.28.3 </details<details</details<details</details<details</details<details</details<details</details
/sig network
/assign
Thank you for this. Please use flag to change metric listening address, docs are here: as for the , the docs are a bit confusing wrt to this field. /kind support
--metrics-bind-address ipport Default: 127.0.0.1: | The IP address and port for the metrics server to serve on, defaulting to "127.0.0.1:" (if --bind-address is unset or IPv4), or "[::1]:" (if --bind-address is IPv6). Set metrics-bind-address to ::1, the effect is as follows tcp6 0 0 ::1: :::* LISTEN /kube-proxy So the document can also be updated. --bind-address may not be useful.
/triage accepted


Question: } } <del> // And create a configuration source that reads from a local file config.NewConfigSourceFile(*configFile, serviceConfig.Channel("file"), endpointsConfig.Channel("file")) glog.Infof("Using configuration file %s", *configFile) </del> loadBalancer := proxy.NewLoadBalancerRR() proxier := proxy.NewProxier(loadBalancer, net.IP(bindAddress)) // Wire proxier to handle changes to services

It's not well tested nor widely used; so it should either be removed or have tests and uses .


Question: onHold bool // proberResultsMetricLabels holds the labels attached to this worker <del> // for the ProberResults metric. proberResultsMetricLabels prometheus.Labels </del> <ins> // for the ProberResults metric by result. proberResultsSuccessfulMetricLabels prometheus.Labels proberResultsFailedMetricLabels     prometheus.Labels proberResultsUnknownMetricLabels    prometheus.Labels </ins> } // Creates and starts a new probe worker.

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.


Question: } } } <ins> removePod := func(i int) operation { return func() { if err := cache.RemovePod(pods[i]); err != nil { t.Error(err) } } } </ins> removePodWithAffinity := func(i int) operation { return func() { if err := cache.RemovePod(podsWithAffinity[i]); err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If we delete node first, then remove pods on this node, the scheduler will crash in function What you expected to happen: Not crashed How to reproduce it (as minimally and precisely as possible): some pods on node A delete node A delete pods on node A Anything else we need to know?: Environment: Kubernetes version (use ): 1.16-1.18, even master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
can you show the core dump please of this crash?
/assign
I patched the unit test shown in into the , and branches. None crashed. I'll await for the crash dump.
Here is the fresh dump.
and which version was this?
This is master branch. And I've tested it from 1.16-1.18, it can be reproduced easily.
Thanks a lot. I was able to finally reproduce in unit tests. The scenario is: a node with at least 2 pods. node. pod1. In unit test, this is first an update to set the termination timestamp. No issues up to this point. pod2. Again, first an update to set the termination timestamp. Here it crashes. I'm working on a fix.
It looks like this is not an issue in 1.17 and below, because we preserved a placeholder node where the operations occur. do you have stacktraces for 1.17 and 1.16?
Yes, it seems only be found at 1.18. :smile:


Question: "k8s.io/client-go/dynamic" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" <ins> "k8s.io/kubectl/pkg/util/podutils" </ins> podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/kubelet" "k8s.io/kubernetes/test/e2e/framework"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: It has been flaking more frequently since 3/13. Testgrid link: Reason for failure: The issue seems to be a timeout cc /kind flake /sig node /priority important-soon /milestone v1.14
Latest PR to touch that test May be due to a readiness probe failing.
/remove-kind failing-test sending this your way for triage too - could you have a look please? I wonder if it's a node problem or if timeouts are originating elsewhere fwiw, it's failing about 10%-15% of the time in e2e tests:
It looks like updating the status after the patch to the readiness gate takes longer than expected.

Looking at this failure: From the test logs: [It] should support pod readiness gates [NodeFeature:PodReadinessGate] ï¿½[1mSTEPï¿½[0m: submitting the pod to kubernetes ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true Mar 14 01:52:56.720: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:57.731: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:58.725: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:59.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:00.721: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:01.708: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:02.753: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:03.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:04.713: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.709: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.767: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false From the kubelet logs: The pod goes to Ready at 1:53:12, and we stopped checking at 1:53:05 This looks expected, based on the description of the parameter in the test. I don't think this should be release-blocking.

/assign who owns this feature to determine how we can make the test less flaky. Feel free to close if this is already tracked somewhere else /unassign
Thanks, good to know. if it could be worked on, it would really help stabilize CI - which would be super appreciated, especially for release-blocking jobs.
When this tests was (2 or 3 release back), I remember the 10 second waiting timeout is enough for kubelet to react and update the pod ready condition. IIRC, kubelet reconciles pod status every 15s. But if pod readiness gate condition is updated, kubelet should reconcile ASAP skipping the 15s period. There might be a bug here.
Thanks - do you have time to take a look? It would help to know (especially if we need to include something in known issues).
I just did a few manual tests. Kubelet reacted very quickly. Not able to reproduce the update latency more than 10s.
I ran the test in a loop more than 50 times now on my own e2e cluster. Not a single failure. Is it because there are many tests running in parallel causing high load on kubelet, hence longer update latency?
Interesting, could well be the case Is there a way to validate that? And, is there something here we should raise in terms of performance/scalability (e.g. "update latency varies with pod load")?
That is a well-known result of the rate limiting of the kubelet. The kubelet can only send 5 qps of updates to the API Server, so the kubelet is limited in its ability to update statuses when there are lots of updates to send, which occurs when there is high pod churn.
This issue seems to not be a problem anymore. /close
Closing this issue. <detailsIn response to : This issue seems to not be a problem anymore. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Should this test be marked [Serial] then? (Or maybe flaky?) running into this again
Can you share the test grid link where it is flaky?
I've been testing extensively myself offline to re-evaluate tests to use in presubmit.

this sounds like inherently this test is not safe to run with lots of other tests in parallel, which would make it a candidate for
/reopen This is the flakiest test for PRs Triage link: Since this is a GA feature it should have conformance tests covering it, this specific test was prevented from promotion to conformance due to flakiness issues called out here One of the comments suggested rewriting to use an informer or poll for longer than 10s, and not use an explicit sleep. Tagging with [Serial] seems like it would work from a brute force perspective. Is the 10s number an important or documented SLO somewhere?
Reopened this issue. <detailsIn response to : /reopen This is the flakiest test for PRs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
another data point of sorts: IIRC I disabled this test in kind because it was too flaky, i wasn't aware that it was also flaky on GCE.
opened to deflake
Still seen after disabling auto flake retry:
top-ten flake according to
Confirming this flake is still top-ten after we merged Thanks for all your work here !
no longer top-ten, but still flaky -
back in the top ten according to
/assign assigning sig-node leads for visibility/routing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /remove-priority critical-urgent /priority important-longterm Not in the top ten, but it still flakes occasionally
I'm pretty sure we still blacklist this test in most kind CI because it's too flaky. Aaron Crickenberger < :
/milestone v1.19
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Gonna push this with sig node /cc /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale It's still failing across a number of jobs but I can't tell that it's as painful a flake today
/milestone clear
/assign /triage accepted
GitHub didn't allow me to assign the following users: fromanirh. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
working to reproduce outside CI - way easier to debug
still working my way towards a reproducer.
has been unable to reproduce locally. We are still seeing occasional timeouts in CI: Since this was the only timeout on that test run, we may want to consider tweaking the timeout thresholds?
I'm happy I have time again to work on this issue. It doesn't reproduce locally on BM, and this seems indeed to point to a too busy worker in CI and/or too short timeout. So let's start with the simplest fix:
Today in , it is the top 1 flake for "pr:pull-kubernetes-e2e-gce-ubuntu-containerd".
ok, time for me to prioritize this work
Do we have a testgrid link? I'm looking at to try to understand better the flakes and getting too little from here (but looks relatively good from here?)

More recent triage link: Appears to be failing in a number of containerd and crio jobs. /unassign /priority important-soon /remove-priority important-longterm
/milestone v1.22
Did we vet that this is fixed after ?


Question: fi if [[ -z "${KUBE_ADMISSION_CONTROL:-}" ]]; then <del> ADMISSION_CONTROL="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,PodPreset,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,Priority,StorageObjectInUseProtection" </del> <ins> ADMISSION_CONTROL="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,Priority,StorageObjectInUseProtection" </ins> if [[ "${ENABLE_POD_SECURITY_POLICY:-}" == "true" ]]; then ADMISSION_CONTROL="${ADMISSION_CONTROL},PodSecurityPolicy" fi

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pull-kubernetes-e2e-gce and pull-kubernetes-kubemark-e2e-gce both start kube-apiserver with admission and but without enabling priorityclasses and podpresets resources. We can see many error logs from
/sig test /sig api-machinery
/kind bug
/sig testing


Question: return append(allErrs, field.Invalid(fldPath, value, "must be a standard resource for containers")) } } <del> return field.ErrorList{} </del> <ins> return allErrs </ins> } // Validate resource names that can go in a resource quota

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In some validation functions, such as , , value returned may be wrong. When return some errors and , we may just return , that is not expected What you expected to happen: return correctly
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/assign
Can you clarify "wrong"?
For example in will add and to field of ResourceQuotaSpec. Before , these two resources are not to standard resource sets, will return error since they are not fully-qualified neither. But will not return any error since the two resources are to standard quota resource name sets. The result above is not expected . Error happened in , we need to return it instead of returning when we ran


Question: kube::util::ensure-gnu-sed kube::log::status "Building kubectl" <del> make -C "${KUBE_ROOT}" WHAT="cmd/kubectl" </del> <ins> make -C "${KUBE_ROOT}" WHAT="cmd/kubectl cmd/kubectl-convert" </ins> # Check kubectl kube::log::status "Running kubectl with no options"

was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release


Question: "//vendor/k8s.io/client-go/discovery:go_default_library", ], ) <ins> go_test( name = "go_default_test", srcs = ["status_strategy_test.go"], embed = [":go_default_library"], deps = ["//vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library"], ) </ins>

the said: However the are different, it does not allow to set spec and metadata, but other changes are not ignored. And updates to other parts are not validated.
/cc for some suggestions.
/assign
Made a pull request:
Is the documentation wrong or the implementation? I think there was a longer discussion about the correct behaviour in the PR, but haven't found it yet.
It strange that other parts rather than metadata and spec can be modified and they are not validated. So I think the implementation is wrong, at least.
We should follow the design doc, I think.
/sig api-machinery
/priority important-soon
I agree the current behaviour is wrong. Added some comments to Especially, worries me whether we really have to right /status behaviour. Not really related to the given issue. But as we are touching the /status behaviour, we should clarify that as well.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 3 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: source "config-default.sh" KUBECTL="${KUBE_ROOT}/cluster/kubectl.sh" export KUBECTL_PATH="${KUBE_ROOT}/cluster/ubuntu/binaries/kubectl" <ins> export KUBE_CONFIG_FILE=${KUBE_CONFIG_FILE:-${KUBE_ROOT}/cluster/ubuntu/config-default.sh} </ins> function init { echo "Creating kube-system namespace..."

This is not a problem when running kube-up because the kube-up for Ubuntu ensures that environment variable is defined. But when an operator runs , kubectl is invoked without ensuring KUBECONFIGFILE is defined and invokes detect-master without ensuring KUBECONFIGFILE is defined.


Question: if addSCTPContainers { protocols = append(protocols, v1.ProtocolSCTP) } <ins> if framework.NodeOSDistroIs("windows") { return NewWindowsModel(namespaces, []string{"a", "b", "c"}, []int32{80, 81}, dnsDomain) } </ins> return NewModel(namespaces, []string{"a", "b", "c"}, []int32{80, 81}, protocols, dnsDomain) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: if serviceNameVal != endpoints.Name { t.Errorf("Expected EndpointSlice to have %s label set to %s, got %s", discovery.LabelServiceName, endpoints.Name, serviceNameVal) } <ins> _, ok = epSlice.Annotations[corev1.LastAppliedConfigAnnotation] if ok { t.Errorf("Expected LastAppliedConfigAnnotation to be unset, got %s", epSlice.Annotations[corev1.LastAppliedConfigAnnotation]) } _, ok = epSlice.Annotations[corev1.EndpointsLastChangeTriggerTime] if ok { t.Errorf("Expected EndpointsLastChangeTriggerTime to be unset, got %s", epSlice.Annotations[corev1.EndpointsLastChangeTriggerTime]) } for annotation, val := range endpoints.Annotations { if annotation == corev1.EndpointsLastChangeTriggerTime || annotation == corev1.LastAppliedConfigAnnotation { continue } if epSlice.Annotations[annotation] != val { t.Errorf("Expected endpoint annotation %s to be mirrored correctly, got %s", annotation, epSlice.Annotations[annotation]) } } </ins> } for _, epSubset := range endpoints.Subsets {

If I modify a custom endpoint, the kubectl annotation is mirrored to the generated endpoint slice Since the kubect last-applied-configuration tracks the status of the object, it shouldn't be mirrored since it doesn't apply to the generated object a custom endpoint an update on any of the fields the generated endpoint slice annotations /sig network
ping
if we agree this is a bug, the fix is just and adding some tests of course :)
Thanks for catching this! Agree this is a bug that we should fix.
/triage accepted
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey, I'm on it!
Hi While I have made the update mentioned in the above comment, I feel a bit lost on where to add/update tests. I understand that I probably shouldn't raise a PR without trying to add a few tests for this issue. So could you possibly guide me to the correct files(s) I should be looking into where the annotations for updated EndpointSlices are being tested?
Hey I think the easiest place to add tests might be here: You could add another test case where the Endpoints object had this label and then ensure that label was not copied over to an EndpointSlice.
Alright I'll check out these tests. Thank you!
/assign


Question: <ins> { "swaggerVersion": "1.2", "apiVersion": "", "basePath": "https://10.10.10.10:6443", "resourcePath": "/apis", "apis": [ { "path": "/apis", "description": "get available API versions", "operations": [ { "type": "void", "method": "GET", "summary": "get available API versions", "nickname": "getAPIVersions", "parameters": [], "produces": [ "application/json" ], "consumes": [ "application/json" ] } ] } ], "models": {} } </ins>  No newline at end of file

Loading returns Looking at the code is registered in the same way as which works fine. So not sure why is this broken.


Question: } certificateARN := annotations[ServiceAnnotationLoadBalancerCertificate] <del> if certificateARN != "" && (sslPorts == nil || sslPorts.numbers.Has(int64(port.Port)) || sslPorts.names.Has(port.Name)) { </del> <ins> if port.Protocol != v1.ProtocolUDP && certificateARN != "" && (sslPorts == nil || sslPorts.numbers.Has(int64(port.Port)) || sslPorts.names.Has(port.Name)) { </ins> portMapping.FrontendProtocol = elbv2.ProtocolEnumTls portMapping.SSLCertificateARN = certificateARN portMapping.SSLPolicy = annotations[ServiceAnnotationLoadBalancerSSLNegotiationPolicy]

What happened: What you expected to happen: kubernetes creating an AWS NLB for UDP How to reproduce it (as minimally and precisely as possible): Create service with annotation and port with Anything else we need to know?: AWS recently announced that UDP is available within their NLB Environment: Kubernetes version (use ): 1.11.9 Cloud provider or hardware configuration: AWS
/sig aws
/kind feature
/remove-kind bug
/milestone v1.16
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.16
I am taking up this Feature Development. How can this be assigned to me?
/assign
+1
+1 I am experiencing the same issue. NLB also supports UDP_TCP type, any service using "nlb" should be able to mix TCP and UDP port configs.
Any progress? I can collaborate on this, if you need a hand.
+1
+1 On Thu, Jul 18, 2019 at 12:58 Adam Murphy <:
+1
I just ran into this. Any updates?
Attempting to fix it with
Thanks, The Changes looks good.
This should be closed I think.
This feature only for 1.16?
Could feature back ported into 1.15 1.14 1.13
Could someone review please?
/sig cloud-provider
Any update on this?
As we are in the code freeze and this issue is still not fixed, I'll move it to the 1.17 milestone. I'm also going to add the PR that fixes this issue to the milestone. If there are any concerns, please ping me or ping SIG-Release on Slack. /milestone v1.17
Do you have a timeline when this will be released?
This is a breaking issue in multiple versions but the is not prioritized. Does anyone have a workaround at the moment?
For AWS you can associate elastic IP with pod. Need make aws cli call.

If you need static private IP then you can create DaemonSet attach NIC with static IP as second NIC to EC2 instance and node some label (like ). Later you start pod with network, get node label, bound to required host IP. If you need static public and private IPs, then you can customize node-group init script and attach NIC with permanent private IP and Elastic IP at node startup. Later simple start pod with network
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
Hi bug triage team for 1.17, will this issue be resolved before code freeze on Nov. 14 (Next Thursday)?
/milestone v1.18 (v1.17 code freeze)
looking for a first issue to work, can I take this
gave some good feedback to the original PR. Does this need a reset with a fresh PR? I also have a need for this. I don't have a strong urge to write, or reuse, an operator for NLBs in AWS for UDP. This will eventually get done in k8s anyways, so I'd rather put the effort there.
BTW, you can always setup an NLB manually to forward UDP to a nodePort service on your k8s instances. The issue with that is that you have to pick a NodePort service for UDP before hand, and manually handle the NLB and the target groups. One cluster upgrade and your target group behind the NLB disappears since all of the instances get replaced. It's not ideal but in a pinch...
I don't mind updating a the PR, seems like a good first issue for me.
/assign
+1
+1
+1
+1
+1
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for milestone 1.18? Thanks in advance!

C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/remove-kind bug
Is there any work remaining to be done on this? Willing to contribute if so, would really love to see this make it into 1.18!
Thanks for the reply This has been a long running issue, and given that there has not been any recent activity, would it make more sense to once again move the milestone? We are 2 days away from code freeze (5 March).
I wonder what "activity" you are expecting.
We are now in code freeze for 1.18, so I am moving this to the next milestone. /milestone v1.19
Hello, as of now, I see is 65% completed. May we please have an update on estimated release date of UDP support for AWS NLB?
People have been trying to be constructive here. But the team doesn't even try to tell us what is pending. Sad.
Would also love to see this feature for EKS - and am willing to help out. Even if we could get it in 1.19 it would still take a long time to become available in EKS (which is currently at 1.15). So the sooner we can start work on this - the better :-)
+1 Also hope we could get in 1.19 and happy to help out!
any update on this pretty please ?
whats the update here?
From what I can see, the main issues that remain unresolved since the first PR last year: is still pending if you need a service that supports both TCP and UDP, e.g., DNS. DNS is also a more-complicated example because it supports TCP and UDP on the same port number, and that'll need more refactoring here, as AWS treats that as a single multi-protocol target. Edit: To be clear, not really a blocker for UDP support, but it's been raised in the discussion earlier. AWS NLB can only do TCP-based health checks (including HTTP and HTTPS), so your service needs to have a health-check TCP port listening. Current PRs silently assume it's on the same port number as the UDP service you're advertising.

If you're using , then it'll use the health check from  instead anyway. So this is probably only an issue for LoadBalancer Services with set. See "Things to know" on the .


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "strings" "sync"

[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining StatefulsetScale endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, apps endpoints sql-mode select distinct endpoint, right(useragent,73) AS useragent from testing.auditevent -- where useragent ilike '%subresource%' where endpoint ilike '%AppsV1NamespacedStatefulSetScale%' and releasedate::BIGINT round(((EXTRACT(EPOCH FROM NOW()))::numeric)1000,0) - and useragent like 'e2e%' order by endpoint limit 30; example endpoint | useragent -----------------------------------------|--------------------------------------------------------------------------- patchAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] readAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] replaceAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] (3 rows) If a test with these calls gets merged, test coverage will go up by 1 point* This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } if oldRunning != config.Replicas { <del> if pods, err := config.Client.Pods(api.NamespaceAll).List(labels.Everything(), fields.Everything()); err == nil { </del> <ins> // List only pods from a given replication controller. if pods, err := config.Client.Pods(api.NamespaceAll).List(label, fields.Everything()); err == nil { </ins> for _, pod := range pods.Items { <del> Logf("Pod %st%st%st%s", pod.Namespace, pod.Name, pod.Spec.NodeName, pod.DeletionTimestamp) </del> <ins> Logf("Pod %st%st%st%s", pod.Name, pod.Spec.NodeName, pod.Status.Phase, pod.DeletionTimestamp) </ins> } } else { Logf("Can't list pod debug info: %v", err)

From time to time, different tests are failing with the error like this: Expected error: <*errors.errorString | 0xc20bcd3a30: { s: "Only 29 pods started out of 30", } Only 29 pods started out of 30 not to have occurred Few examples:
I didn't see this happening for ~month now. Closing this - feel free to reopen if you observe it again.


Question: StorageClassName: &classGold, }) <ins> validClaimWithNonIntegerStorage := validClaim.DeepCopy() validClaimWithNonIntegerStorage.Spec.Resources.Requests[api.ResourceName(api.ResourceStorage)] = resource.MustParse("1001m") validClaimByStorageClassWithNonIntegerStorage := validClaimByStorageClass.DeepCopy() validClaimByStorageClassWithNonIntegerStorage.Spec.Resources.Requests[api.ResourceName(api.ResourceStorage)] = resource.MustParse("1001m") </ins> evaluator := NewPersistentVolumeClaimEvaluator(nil) testCases := map[string]struct { pvc   *api.PersistentVolumeClaim

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When a pvc was specified with a quantity including the suffix : attempts to include that quantity in the resourcequota's storage resulted in the following error, which blocked all other updates to the resourcequota: What you expected to happen: Updates to all other quantities in the resourcequota should not have been blocked by the error above In general, as a suffix (milli) doesn't really make sense when used in the context of memory, and causes issues like the above. Would it be possible to deny the usage of this suffix at admission time for things like pvcs? (Our mitigation right now is to add a rule that does this in Open Policy Agent). How to reproduce it (as minimally and precisely as possible): Create a resourcequota object with limits on pvcs and the a storage type, e.g. Then create a PVC with the following spec: The resource quota controller should start producing the error log above. Anything else we need to know?: Environment: Kubernetes version (use ): v1.17.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/remove-sig api-machinery /sig storage
i guess we should leave api-machinery! /sig api-machinery
/assign I think we should validate the value of field.
I've filed to address this issue.
looks like it is not easy to tighten the validation and that requires careful consideration xref:
/assign
"Updates to all other quantities in the resourcequota should not have been blocked by the error above" should be fixed separately
Opened to round storage requests up to the nearest byte in the quota evaluator.


Question: } for _, ns := range namespaces { <del> framework.RunKubectlOrDie(ns.Name, "create", "-f", backendRcYaml, getNsCmdFlag(ns)) </del> <ins> framework.RunKubectlOrDieInput(ns.Name, read(filepath.Join(clusterDnsPath, "dns-backend-rc.yaml")), "create", "-f", "-") </ins> } for _, ns := range namespaces { <del> framework.RunKubectlOrDie(ns.Name, "create", "-f", backendSvcYaml, getNsCmdFlag(ns)) </del> <ins> framework.RunKubectlOrDieInput(ns.Name, read(filepath.Join(clusterDnsPath, "dns-backend-service.yaml")), "create", "-f", "-") </ins> } // wait for objects for _, ns := range namespaces { <del> e2eresource.WaitForControlledPodsRunning(c, ns.Name, backendRcName, api.Kind("ReplicationController")) e2enetwork.WaitForService(c, ns.Name, backendSvcName, true, framework.Poll, framework.ServiceStartTimeout) </del> <ins> e2eresource.WaitForControlledPodsRunning(c, ns.Name, backendName, api.Kind("ReplicationController")) e2enetwork.WaitForService(c, ns.Name, backendName, true, framework.Poll, framework.ServiceStartTimeout) </ins> } // it is not enough that pods are running because they may be set to running, but // the application itself may have not been initialized. Just query the application. for _, ns := range namespaces { <del> label := labels.SelectorFromSet(labels.Set(map[string]string{"name": backendRcName})) </del> <ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"name": backendName})) </ins> options := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(ns.Name).List(context.TODO(), options) framework.ExpectNoError(err, "failed to list pods in namespace: %s", ns.Name) <del> err = e2epod.PodsResponding(c, ns.Name, backendPodName, false, pods) </del> <ins> err = e2epod.PodsResponding(c, ns.Name, backendName, false, pods) </ins> framework.ExpectNoError(err, "waiting for all pods to respond") framework.Logf("found %d backend pods responding in namespace %s", len(pods.Items), ns.Name) <del> err = waitForServiceResponding(c, ns.Name, backendSvcName) </del> <ins> err = waitForServiceResponding(c, ns.Name, backendName) </ins> framework.ExpectNoError(err, "waiting for the service to respond") }

What happened: We've been doing some release readiness and auditing all tagged ginkgo tests... Looks like The test seems to expect a GOPATH when running. Im seeing some failures in sonobuoy and suspect a possible relation there but havent dove into it. Either way, its clear this test might need some updating/fixing/cleanuping and parts can be rewritten/improved. What you expected to happen: No reliance on GOPATH for an e2e test ... I dont remember the whole history on this, but no tests that rely on static files currently look in gopath, from what i can tell ? instead we plumbed in a repo-root/ option a few years ago. There may be a state of the art in terms of how we do this.... but i thought it was at one point which was an option passed in. How to reproduce it (as minimally and precisely as possible): Try to run this in sonobuoy or any other environment where there's no GOPATH or non-standard source trees, pretty sure it might not work. Anything else we need to know?: I think the fix here is to just embed the YAML in golang like we do in other tests. along the way there's other cleanup we can do to improve this test. Environment: Kubernetes version (use ): master
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
the YAML files are pretty small, so +1 to embed them in i don't think testing the YAML from the separate repository () directly is required here. ref /help /kind cleanup /priority backlog
/remove-help
/assign Also I should say, I talked to Jay on slack and he is ok with me taking a look at it, if thats ok.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
xref: go 1.16 has built in support for embedding go 1.17 looks to be removing GOPATH
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/help wanted
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still a problem. FYI re: GOPATH removal
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/assign


Question: # OS options for minions KUBE_OS_DISTRIBUTION="${KUBE_OS_DISTRIBUTION:-jessie}" <del> KUBE_MASTER_OS_DISTRIBUTION="${KUBE_OS_DISTRIBUTION}" KUBE_NODE_OS_DISTRIBUTION="${KUBE_OS_DISTRIBUTION}" </del> <ins> MASTER_OS_DISTRIBUTION="${KUBE_OS_DISTRIBUTION}" NODE_OS_DISTRIBUTION="${KUBE_OS_DISTRIBUTION}" </ins> KUBE_NODE_IMAGE="${KUBE_NODE_IMAGE:-}" COREOS_CHANNEL="${COREOS_CHANNEL:-alpha}" CONTAINER_RUNTIME="${KUBE_CONTAINER_RUNTIME:-docker}"

I think got confused between e.g. KUBEMASTEROSDISTRIBUTION and MASTEROS_DISTRIBUTION; it definitely doesn't match GCE. PR coming in once I've tested.
cc


Question: } } <ins> func (cnc *CloudNodeController) UpdateCloudNode(_, newObj interface{}) { if _, ok := newObj.(*v1.Node); !ok { utilruntime.HandleError(fmt.Errorf("unexpected object type: %v", newObj)) return } cnc.AddCloudNode(newObj) } </ins> // This processes nodes that were added into the cluster, and cloud initialize them if appropriate func (cnc *CloudNodeController) AddCloudNode(obj interface{}) { node := obj.(*v1.Node)

Once cloud-controller-manager is started we should remove the taint. right?
/sig cluster-lifecycle
/area cloudprovider do you have anymore information on this? I can't reproduce this on existing nodes before the CCM was created or new nodes to a cluster.
i can recreate this using by setting along with a CLOUDCONFIG + CLOUDPROVIDER env vars
If you're running a local cluster then how are you running the CCM? The only way the taint gets removed is if the instance was found in the cloud provider and setup (labels, addresses, etc) OK. The code can be found
the local cluster is started within a openstack vm. so it's a one vm cluster. thanks for the pointer, will check it out
/assign
- So ... when i use local-up-cluster, the kubelet starts up before the cloud-controller-manager. So the shared informer that calls is never triggered. This patch seems to fix the issue. Note that RBAC stuff needed to be setup for as well for the to succeed.


Question: default: } <del> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { </del> <ins> obj, err := decodeListItem(ctx, data, uint64(kv.ModRevision), s.codec, s.versioner, newItemFunc) if err != nil { </ins> recordDecodeError(s.groupResourceString, string(kv.Key)) return err } <ins> // being unable to set the version does not prevent the object from being extracted if matched, err := opts.Predicate.Matches(obj); err == nil && matched { v.Set(reflect.Append(v, reflect.ValueOf(obj).Elem())) } </ins> numEvald++ // free kv early. Long lists can take O(seconds) to decode.

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return p.schedulingCycle } <del> // AddUnschedulableIfNotPresent does nothing if the pod is present in any // queue. If pod is unschedulable, it adds pod to unschedulable queue if // p.moveRequestCycle > podSchedulingCycle or to backoff queue if p.moveRequestCycle // <= podSchedulingCycle but pod is subject to backoff. In other cases, it adds pod to // active queue. </del> <ins> // AddUnschedulableIfNotPresent inserts a pod that cannot be scheduled into // the queue, unless it is already in the queue. Normally, PriorityQueue puts // unschedulable pods in `unschedulableQ`. But if there has been a recent move // request, then the pod is put in `podBackoffQ`. </ins> func (p *PriorityQueue) AddUnschedulableIfNotPresent(pod *v1.Pod, podSchedulingCycle int64) error { p.lock.Lock() defer p.lock.Unlock()

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: hook should execute prestop http hook properly [NodeConformance] [Conformance]' description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. <del> A server pod is created that will serve http requests, create a second pod with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </del> <ins> A server pod is created that will serve http requests, create a second pod on the same node with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </ins> release: v1.9 file: test/e2e/common/node/lifecycle_hook.go - testname: Container Runtime, TerminationMessage, from log output of succeeding container

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work.

As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?

oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:


Question: if err != nil { return nil, err } <del> serviceTags, err := getServiceTags(service) if err != nil { return nil, err } </del> <ins> serviceTags := getServiceTags(service) </ins> var sourceAddressPrefixes []string if (sourceRanges == nil || servicehelpers.IsAllowAll(sourceRanges)) && len(serviceTags) == 0 { if !requiresInternalLoadBalancer(service) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider has a limited support for service tags by annotations: But according to , there are actually much more service tags than this list. Hence, errors would happen if the service tag is not included in the above list. What you expected to happen: Azure cloud provider should support all service tags from Azure. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig azure /priority important-soon
/assign